Three cheers for research users engaging in REF 2014

In the UK, research outputs from universities are assessed every five years to determine future funding allocations from government. In 2014, for the first time, the Research Excellence Framework (REF) included an assessment of research impact. This component was worth 20 per cent of the score awarded to each institution. The assessment was performed by panels of academics and research users.

The importance of research user engagement throughout the process of REF 2014 cannot be overstated. Research users, or those benefitting from publicly funded university research, played key roles throughout REF 2014 in several ways. Their evidence was needed to substantiate academics’ claims about the wider impact of university research – conveyed through impact case studies and strategies. Secondly, representatives from beneficiary organisations, like the British Library, the Overseas Development Institute, the BBC, Royal Museums Greenwich, Oxfam, BT, BAE Systems and the Bank of England served on the panels that assessed the impact of university research. So their role in engaging with REF 2014 has been vital to its success and important to the future of the process. Continue reading

Who was hacked? A New Report Investigates

Dr Martin Moore is the author of ‘Who was hacked? An investigation into phone hacking and its victims. Part 1 – The News of the World’. He is director of the Media Standards Trust and a Visiting Senior Research Fellow at King’s College London.

Read coverage of the various hacking trials (News of the World in 2013/14 and now Mirror Group Newspapers) and you could be forgiven for thinking phone hacking was all about celebrities. Celebrities attract attention, attract news interest, and sell papers. This is perhaps why many of the news reports of phone hacking have concentrated on celebrities and are illustrated with photographs of celebrities.

Yet, if you actually sit down and add up the numbers, it becomes clear that though many celebrities were targets of the News of the World they were not the main victims of phone hacking. Over two thirds of the News of the World phone hacking victims that we know about were not public figures. They were beauticians, receptionists, lawyers, estate agents, nannies, policemen, journalists, priests, sports agents and hairdressers. Continue reading

Influential, international and interdisciplinary: The impact of the UK’s research

For the first time, Higher Education Institutions (HEIs) were required to submit impact case studies as part of the 2014 Research Excellence Framework (REF). In total, 6679 non-redacted case studies were submitted, and today, we publish a report of the results of our text mining analysis of these data.

The case studies are now available to read online in a searchable database developed by Digital Science, providing a rich resource that has enabled us to demonstrate that UK research has thousands of different applications worldwide. The analysis of the case studies, led by the Policy Institute and department of digital humanities at King’s College London, used text mining techniques leading to the identification of 60 impact topics or areas where research influences society, such as medical ethics, climate change, clinical guidance and women, gender and minorities. Automated text mining was also supplemented with ‘deep mines’, where more than 1000 case studies were read to provide a deeper picture of the data – looking at specific questions such as ‘what is the impact and value of research on clinical practice and health gain?’ and ‘what has been the impact of research on BRIC countries?’.

The results of this analysis are fascinating and also discussed in Research Fortnight and on the HEFCE blog. The take home message is that the benefits from research are multi-impactful. For example, the case studies submitted for the unit of assessment on psychology, psychiatry and neuroscience showed that these disciplines made a contribution to 49 of the 60 impact topics. These included obvious applications such as in mental health, but also in impact topics including transport, schools and education and crime and justice. We also discovered that the case studies contained more than 3700 individual pathways to impact presenting a real challenge to anyone interested in producing impact metrics.

Many of the case studies were able to provide a clear illustration of the contribution that universities make to society, in a way that has not been revealed before. For example, one case study reported on research showing that the painkiller co-proxamol was the most common drug used for suicides in the UK. This finding led to its withdrawal, and has been estimated to have led to approximately 600 fewer deaths by 2012 in the UK alone.

Our analysis has also shown that impact of UK universities is truly global with the Details of the Impact sections of the case studies mentioning every country in the world – with the US being the most frequently mentioned, followed by Australia, Canada and Germany.


The global reach of impacts arising from research undertaken in UK HEIs

The case studies provide a rich resource demonstrating the breadth and depth of research impact. They can also help us to change perceptions. For example, the largest impact topic was informing government policy, which was associated with 1233 case studies. The word ‘policy’ was mentioned in 3206 case studies. This is reassuring, given the pre-REF scepticism about whether case studies could capture and articulate impacts on public policy. It was suspected that researchers mostly influenced policy through personal contacts and under-the-radar advisory channels, rather than through specific research that could be described in a case study.

Our analysis is really just the start. There are limitations to using these case studies as research material, such as the universally positive sentiment in their language and the fact that researchers could carefully select which case studies to use and the number of identical or near-identical submissions. However, this rich resource provides numerous avenues for future research – to enable us to really dig deeper into the global impact of UK University research.

Dr Saba Hinrichs, Senior Research Fellow & Professor Jonathan Grant, Director, The Policy Institute at King’s

PolicyWonkers in Research and Development: Time to go to school?

Investing in R&D is seen as key to contributing to a healthier population, better environment and greater socio-economic prosperity. The impacts of these investments need to be measured and assessed to inform policy requirements, and policymakers in governments around the world are seeking such evidence. However, how do we know where to build capacity to be able to provide this evidence especially given the methodological challenges involved, such as long timelines, cause and effect attribution and getting the right data? How do we deal with the unravelling dynamic and serendipitous nature of R&D itself?

Five years ago, those interested in learning about the assessment of the impact of R&D programmes could find limited scientific articles in PubMed and institutional reports on Google, but it was difficult to locate international experts and learn from their experiences.

There were great champions pushing forward important developments. Led by visionaries such as Martin Buxton and Stephen Hanney from the HERG Group at Brunel University, who developed the Payback model to evaluate the returns of R&D, and Jonathan Grant and Steven Wooding from King’s College London and RAND Europe who developed and implemented path breaking studies, the UK was an oasis in the international desert. Canada was another oasis, led by passionate advocates such as Cy Frank who chaired a panel of the Canadian Academy of Health Sciences that proposed a ready-to-use framework and indicators that was instrumental in promoting research impact assessment (RIA) best practices in Canada and abroad. Cy was a great man who sadly passed away two weeks ago and will be missed by many.

More needed to be done and the real need for mutual learning (from the doers of RIAs and the policymakers who use the results to inform policies) was the impetus for creating a new oasis in 2013 – the International School on Research Impact Assessment (ISRIA).

isriaISRIA’s mission is to advance knowledge, to build global capacity across all fields of science and to promote an international community of practice in RIA. The school has a number of principles and values, including an agnostic approach in not advocating for one approach over another, and an open and accessible platform for the community to access tools and learning resources that are oriented to deliver social value and provide practical, feasible and cost-effective solutions. The School is a gathering place for people to share a common language of research impact, share emerging and best practice, learn how to use the tools of the trade and move the science of science forward in a practical sense. It is run over five days and focuses on teaching how to assess impact of R&D as well as how best to communicate results to inform policy.

The school is hosted annually in a different continent with the spirit of incorporating global perspectives to address local needs. The inaugural school was held in Barcelona, Spain in 2013, and participants from 17 different countries attended.


The second school was held in Banff, Canada in 2014, doubling the number of participants. The school was oversubscribed and in order to meet demand Alberta Innovates – Health Solutions (co-founder and organizer of the School) has established local courses based on ISRIA curricula that will be run annually to as long as the need exists.

isria-people2isria2Since ISRIA was created, more than 200 people have been connected, developed their own RIA plan and shared their implementation experiences. Alumni of previous schools also participate in future events (including a new ISRIA workshop in The Netherlands in collaboration with the Rathenau Instituut), sharing their experiences of the implementation of their RIA plans.

If you are interested in being part of this community of practice, please join the Linked-in group called ‘The international school on research impact assessment’ or send an email to notifying your interest.

The next ISRIA School will be held in Doha, Qatar on 8-12 November 2015. Registrations are now open, and R&D policywonkers around the globe are welcome to attend!

We hope to see you in Doha!

Kathryn Graham
Steering Committee of ISRIA
Executive Director of Performance Management and Evaluation
Alberta Innovates – Health Solutions, Canada

Paula Adam
Co-chair the Steering Committee of ISRIA
Responsible for Research Assessment
Catalan Agency of Health Quality and Assessment, Spain
@PaulaAdam4 @AQuAScat

Blog AquAS in English

A seven-horse race? New research suggests minor parties are being squeezed out of campaign coverage

When broadcasters revised the format of the TV leaders’ debates to include not four but seven political parties, it was a reaction to the UK’s fast changing political landscape. The original line-up included the three established major parties – Conservative, Labour and Liberal Democrat – as well as UKIP after being elevated to ‘major party’ status by broadcast regulators during the EU election campaign. But Prime Minister David Cameron was quick to challenge this composition, arguing that if UKIP should be included so should other minor parties, such as the Greens. Worried about the prospect of ’empty chairing’ the Prime Minster and being open to possible legal challenge, broadcasters soon responded by not only including the Greens, but the SNP and Plaid Cymru. The decision gave credence to predictions that the 2015 general election would see the end of the UK’s longstanding two-party system and witness a rise in the fortunes of the so-called minor parties. Since a hung parliament is predicted to be the most likely outcome, including the possible leaders of potential coalition-building parties in the high profile TV debates seems a sensible way of allowing voters to better understand the choices they face on election day.

Two party squeeze

However, new research from the Media Standards Trust suggests that, far from the media reflecting the full range of political views competing to gain power at Westminster, eleven weeks prior to the general election the minor parties are being squeezed out of coverage. As Figure 1 illustrates, campaign coverage between 16-22 February in the most popular online news providers in the UK – from the BBC, to the Guardian, Daily Mail and Buzzfeed (see here for full sample and method) – overwhelmingly focused on the Conservative and Labour party.

Blog post fig 1 - 04.03.15









In the 1,691 articles featuring parties that will appear on the TV leaders’ debates, the Conservatives appeared in almost three-quarters of coverage (73%) whereas Labour appeared in well over half (55.9%). Despite the party’s incumbency bonus, the Liberal Democrat’s shared equal prominence with UKIP (approx. 17%). The Greens, by contrast, featured in just 5% of articles examined – less the SNP (8.4%) but substantially more than Plaid Cymru (less than 1%).

As Table 1 demonstrates, this imbalance was replicated across all news sources included in the analysis.  With only a couple of exceptions (Buzzfeed and the Daily Star, both of which published a small amount of articles referencing political parties), the parties split into three tiers based on the volume of articles in which they featured. Labour and the Conservatives typically featured in between one-half and three-quarters of articles in which any party was mentioned. The Liberal Democrats and UKIP appeared much less often – usually between 10% and 20% of articles. The remaining minor parties – SNP, Greens and Plaid Cymru – featured considerably less often, and in some news outlets not at all.

Con Lab LD UKIP SNP Greens PC
Daily Mail (N = 316) 78% 51% 18% 12% 5% 4% 1%
Guardian (N = 172) 71% 66% 21% 13% 12% 7% 1%
FT (N = 49) 71% 63% 29% 14% 6% 2% 0%
Mirror (N = 98) 78% 62% 17% 17% 1% 3% 0%
Sun (N = 77) 66% 60% 10% 19% 4% 1% 0%
Times & ST (N = 145) 79% 58% 15% 17% 6% 4% 1%
Telegraph (N = 203) 75% 57% 18% 18% 13% 6% 0%
Independent (N = 150) 70% 55% 14% 23% 9% 8% 1%
Express (N = 98) 79% 42% 11% 18% 13% 2% 0%
Daily Star (N = 23) 61% 35% 9% 26% 0% 4% 0% (N = 147) 61% 52% 24% 14% 18% 7% 1% (N = 32) 53% 56% 13% 13% 9% 0% 0% (N = 13) 85% 31% 8% 8% 8% 8% 0% (N = 48) 81% 52% 15% 13% 10% 6% 0%
Buzzfeed UK (N = 14) 57% 71% 7% 43% 21% 0% 7%
HuffPo UK (N = 106) 73% 64% 20% 25% 1% 10% 0%
Table 1: Party coverage by publication
(percentage of all articles containing references to one or more parties)

It is, of course, difficult to reach any clear conclusions about election coverage based on a week’s analysis. But the Media Standards Trust’s cumulative analysis of coverage over seven weeks reinforces the Conservative and, to a lesser extent, Labour dominance of campaign reporting (see Figure 2).

For the seven week period between January 5 and February 22, the balance of coverage between the seven parties was broadly similar to the seven-day period shown in Figure 1. Once again, the Conservatives dominated, appearing in over three-quarters of articles (75.8%) featuring any party, while references to the Labour Party made up over half of coverage (55%).  The Lib Dems featured in just over one-fifth of news (21.6%), and the UKIP figures show that Week 7 was a good one for them – on average, they appeared in just 14.6% of articles in Weeks 1-7. While the aggregate figures for the three minor parties – the SNP, Greens and Plaid Cymru – were slightly better than in Week 7, they still appeared far less than the two major parties.

Figure 2

Figure 2

Cameron the most visible leader

Since political parties today are increasingly defined by their leaders – exacerbated by the significance paid to the TV debates – their prominence in coverage can reveal an agenda setting power. When isolating the reporting of the seven leaders taking part in the TV debates between 16-22 February (Figure 3), David Cameron alone appeared in almost two-thirds (63.3%) of articles in which any leader was featured, whereas Ed Miliband appeared in roughly a third less than the Prime Minster (39.7%)

Figure 3

Figure 3

Beyond the two major parties, Nigel Farage (17.2%) appeared to a far greater extent than Nick Clegg (8.2%), reinforcing the widespread perception that UKIP’s electoral success lies in its media savvy leader. While Nicola Sturgeon’s media prominence (4.9%) was not far behind Clegg, the Green Party’s Natalie Bennett and Plaid Cymru’s Leanne Wood just managed to scrape 1% between them.

Needless to say, over the next ten weeks or so parties – and their leaders – will enjoy highs and lows in media coverage. Natalie Bennett’s car crash interview on LBC radio last week (shortly after the period analysed by the Media Standards Trust), for example, temporarily pushed the Greens up the campaign agenda. But the episode was short lived as debates about cash for access between Labour and Conservative took centre stage. Moreover, while many candidates will face awkward moments and campaign set- backs, it is arguably the daily drip of news about parties’ policy positions and their credibility in election reporting that will most influence the electoral outcome.

In this respect, according to both the Media Standards Trust’s snapshot weekly picture of coverage as well as the cumulative seven week analysis, the opportunities for the minor parties to pitch their polices and layout their vision for the future appear limited. While the latecomers to the TV debates – the SNP, Plaid and Greens – can use this platform to even up the contest, in day-to-day coverage they are currently being squeezed out of the campaign on the most popular UK online news sites. Far from the 2015 general campaign turning into a seven-horse race, the longstanding two-party dominance of Conservative and Labour remains firmly intact.


Stephen Cushion is a Senior Lecturer at the School of Journalism, Media and Cultural Studies, Cardiff University

Gordon Neil Ramsay is a Research Fellow at the Media Standards Trust and a Visiting Research Fellow at the Policy Institute at King’s College London

Originally posted on Total Politics on 2 March 2015