A competitive landscape for providing services is an opportunity for evidence-building.. but who will benefit?

In a tight fiscal climate the UK government has been an innovator in ways to fund and deliver effective public services. New funding mechanisms, including social impact bonds (SIBs) and other forms of ‘payment by results’, aim to broaden the market for provision of public services and reduce financial risk by ensuring the government does not pay for ineffective services (Cabinet Office 2010).

These new funding arrangements for public services present opportunities for developing the evidence base for policy and practice, since they are associated with different types of organisations delivering services, including for-profit private companies, charities and public-private partnerships. Such providers may also offer new ways of addressing the complex social needs of service users. These approaches to service provision can be seen as a series of experiments. Understanding how these are implemented, what they provide and the outcomes they achieve provides fertile ground for building the evidence base across a range of policy areas. Further, these new funding systems tend to involve collecting data about outcomes for service users and the effectiveness of services, and these data could be used in evaluations and research.

As use of new funding arrangements grows in the UK and internationally, it is worth considering not only the opportunities for evidence-building, but also what might inhibit this.

Competing for improved outcomes, creating better data

Providers competing for contracts to deliver public services must be able to demonstrate their success at improving outcomes for service users. Once contracts have been won, providers need to achieve the best results they can to secure the maximum available payments, and to enhance their chances of being re-commissioned. There is a commercial incentive for providers to monitor and evaluate systematically what they do, reflect on what it is about their intervention that ‘works’, and make improvements where possible. For some this can be transformative. RAND’s research on the SIB at HMP Peterborough and the Drugs and Alcohol Payment by Results pilots include examples where providers have developed new systems for collecting data about the services they deliver and are adopting approaches informed by those data.

In addition to this opportunity for building the evidence base, some of the risks of payment by results schemes have been widely discussed. For example, these funding arrangements may encourage providers to focus too exclusively on an overly narrow set of outcomes– those to which payments are attached and which are most readily quantified (see Disley et al 2011, p. 31- 36).  While including these narrow outcome measures is useful, the data collected about any intervention should also be broad enough to allow detection of benefits beyond the main intended outcomes. And a wider look could also help spot possible unintended negative outcomes.

But there is another risk that has been less extensively debated, and it is one that could impede building the desired evidence base. That risk stems from the new competitive landscape for service provision itself.

Private ownership of the public’s outcomes?

When things go well, competition encourages providers to understand exactly what they’re doing to deliver improved outcomes. But competition could also mediate against sharing successful approaches. Researchers, analysts and policy officials may consider information about what services are provided, to whom, how frequently, and in what combinations, as a shared resource for building a broader evidence base. Yet providers operating in a competitive market may feel a need to hold close such information, in order to retain their commercial edge.

One way in which this risk could be mitigated is for the contractual arrangements behind payment by results to specify how data generated should be shared and used. For example, contracts might specify where the data should be deposited and who is responsible for oversight. Contractual terms also could specify whether it is the commissioner of the intervention who owns the data or some other body, such as the UK’s new national evidence centres. All monitoring and evaluation data could be publically available at the end of any initial trial. These data then could be subject to wider scrutiny before further roll out, informing broader policy and practice and including assessment of positive and negative unintended consequences.

Questions about ownership of the public’s data have wider currency. For example, there is heated debate about who should own and benefit from big data and information collected through people’s online activity. In the May 2014 White House report on big data ownership entitled Big Data: Seizing opportunities, Preserving values, the authors warn that ‘we are building the future we will inherit’. This is true with respect to information and the evidence base on outcomes of public services as well as big data. And if this balance is not found, government departments, researchers and users of services will face a frustrating and detrimental paradox. The quantity and quality of data gathered about interventions and their outcomes may be better than ever, but access to that data will be more restricted and less useful to the very public on whose outcomes it is built.

Jennifer Rubin is Director and Emma Disley is Research Leader in the Communities, Safety and Justice Programme at RAND Europe.

Analysing the Network of an ISIS Twitter Account

It is well documented that the Islamic State of Iraq and al-Sham (ISIS) has an active and well-developed media presence, especially on social networks. Recently, Robert Hannigan, director of the Government Communications Headquarters (GCHQ), wrote in the Financial Times that social networks have become the “command-and-control networks of choice for terrorists and criminals”.

We tested the conclusion reached in other studies that far from being a tightly controlled mouthpiece for ISIS propaganda, Twitter’s structure has resulted in a fragmented network of fighters, sympathisers and disseminators competing for attention. We used open-source tools and techniques to visualise and analyse an ISIS Twitter network centred around a search of mentions and replies for one particular account, @ISTimes2, on the 11th November 2014.*

NodeXL was used to scrape and structure the information from Twitter with the resulting file exported into Gephi, an open-source network visualisation platform. This research worked within certain constraints. Notably, Twitter imposes “rate limiting” when accessing its application programming interface (API) and so a Twitter network based on 1,000 mentions and replies (but excluding follows) was extracted to minimise its impact.

Screenshot 1 shows the extracted network with distinct communities forming in clusters around important nodes. Force Atlas was selected as the layout with average path length and betweeness centrality used to refine the network. Communities were identified using the modularity function which resulted in a network comprised of three main communities.

The community coloured purple is formed around the @ISTimes2 account with the network of users forming a spiral around this central node. The orange community is centred on another important ISIS account, @ISL103. Notably, it has relatively few links to the ISTimes2 network with just six accounts visibly acting as a bridge between the two communities. Finally, the third community, coloured blue, is more diffuse with a multiple important nodes around which the network has formed. These critical nodes include influential and emerging ISIS disseminators such as @ShamiWitness@NusantarWitness and @ShamBreaking3.

Screenshot 1: The complete network with three clearly identified communities

Screenshot 1: The complete network with three clearly identified communities

Screenshot 2 shows the community centred on the @ISTimes2 account with the community of users spiralling around this node. The majority of these users are ISIS supporters but this community also includes a number of journalists and opponents of ISIS engaging with other users.

Screenshot 2: The community centred on the @ISTimes2 account.

Screenshot 2: The community centred on the @ISTimes2 account.

Screenshot 3 shows the @ISL103 account and the largely distinct community that has formed around it.

Screenshot 3: The community formed around the @ISL103 account

Screenshot 3: The community formed around the @ISL103 account

Screenshot 4 shows the third identified community. Unlike the other two it is more diffuse and based around multiple influential disseminators of ISIS propaganda and messages. These accounts have multiple links to the members of the @ISTimes2 community reflecting greater interaction through Twitter mentions and replies.

Screenshot 4: The diffuse third community centred on important ISIS disseminators

Screenshot 4: The diffuse third community centred on important ISIS disseminators

All three communities were composed of very similar members often including a mixture of ISIS fighters (and those claiming to be fighters), sympathisers and disseminators. The majority of the accounts identified in this network analysis were therefore broadly supportive of the message and actions of ISIS. However, a minority of nodes were Twitter users opposed to the group and their ideology. For example @ShameSheep, a user who engages with ISIS supporters and attacks their beliefs and claims.

This network visualisation of the communities interacting with @ISTimes2, confirms findings from prior studies. It demonstrates that ISIS lacks formal centralised control over the communications activities of its fighters and supporters. Instead it exerts indirect control through spreading propaganda messages, often in video or image form that are then shared by its supporters and disseminators to a wider audience. Therefore, although Robert Hannigan is right to identify social networks as crucial elements in the command and control structure of ISIS; the group’s leadership lack the ability to directly control the communication of their ideology.

This analysis also shows that the ability of ISIS to communicate beyond its narrow audience of supporters is fundamentally reliant on disseminators such as @ShamiWitness. As our colleagues at the International Centre for the Study of Radicalisation and Political Violence (ICSR) noted in #Greenbirds a significant proportion of foreign fighters consume information regarding the conflicts in Iraq and Syria from disseminators rather than exclusively through official ISIS channels.

Finally, the ever-shifting landscape of ISIS twitter accounts presents a number of challenges to analysing their networks of communication. Twitter routinely suspends and deletes the accounts of ISIS users for breaching the terms and conditions of the site while ISIS fighters and sympathisers delete their accounts in periodic bouts of paranoia and suspicion. Network analysis can therefore only provide a static glimpse of how these users communicate between themselves and to the outside world. Nevertheless, it provides valuable insights into the diffuse nature of ISIS’s Twitter presence and the importance of disseminating nodes around which communities form.

*Note: many of the Twitter accounts referred to in this piece post graphic text, images and videos.

Post by Mick Endsor, Research Assistant, International Centre for Security Analysis (ICSA). Originally posted on ICSA’s blog – 14 November 2014

Old wine in new bottles? How innovative is the National Security Council?

Dr Joe Devanny, Research Associate at the Institute for Government and the Policy Institute at King’s

Billed as a break with the recent past of opaque, ‘sofa’ government, the National Security Council (NSC) was one of the coalition government’s first creations in May 2010. Its regular meetings, chaired by the prime minister, have brought together senior ministers and top officials to discuss the full spectrum of national security issues, ranging from foreign and defence policy to intelligence and civil contingencies. Several officials with direct experience of the NSC process have praised it for improving the clarity and accessibility of the decision making process.

But just how new is the NSC? That’s a question we try to answer in our new report, The National Security Council: national security at the centre of government. If our perspective extends only to the dawn of New Labour in 1997, then the NSC does indeed look different in its composition, the frequency of its meetings and in sustained prime ministerial commitment to coordinating national security through this formal, central committee process. But if we adopt a longer view, we see many connections between the NSC and its predecessors.

Like its predecessors, the NSC is a cabinet committee, albeit with a different name. Its longest-running predecessor was the Overseas and Defence Cabinet Committee, though many prime ministers have resorted to actual, or de facto, war cabinets to handle military crises or wars.

It is difficult to believe today, but prior to the First World War the Cabinet lacked a secretariat to support its meetings, write and disseminate minutes throughout government. Ministers could leave Cabinet meetings with little idea of what, if anything, had been decided.

National security was the first area in which the need for secretariat support was recognised. The Committee of Imperial Defence (CID) was created in 1902 and augmented with a permanent secretariat in 1904 to coordinate national security more systematically. It was chaired by the prime minister and brought together ministers and senior serving military officers.
During Lloyd George’s premiership, the CID secretariat not only provided the support for his War Cabinet but also formed the model for the Cabinet Secretariat, the precursor to today’s Cabinet Office.

The CID secretary, Maurice Hankey, was the first (and is still the longest serving) Cabinet Secretary. Hankey was a kind of proto-National Security Adviser and Cabinet Secretary combined, a combination which bequeathed significant national security responsibilities to his successors as Cabinet Secretary. Indeed, successive Cabinet Secretaries have been important advisers to the prime minister on security issues, especially during the Cold War era on nuclear and intelligence issues.

Indeed, for many prime ministers, their choice of advisers was more important than the committee structure. Much depends on whether prime ministers feel content with the service they receive from departments: if they find support lacking, or they don’t trust departments or ministers, they can enhance central capacity.
Following the Falklands War, Margaret Thatcher boosted the foreign policy capacity inside Downing St – which had consisted of one foreign affairs private secretary – by employing a senior Foreign Policy Adviser.

The longest serving occupant of this role was Sir Percy Cradock, who combined it with the Chairmanship of the Joint Intelligence Committee (JIC). Foreshadowing the Butler Inquiry’s criticisms of the relationship between policy and intelligence in the early 2000s, Cradock was aware that his dual policy and intelligence roles existed in a certain tension, but he felt nonetheless that the Assessment Staff was an enabler, without which he would have been much less effective and useful to the prime minister.

Though she increased central capacity, Thatcher took decisions with ever smaller circles of advisers. John Major reversed both trends, reverting to a leaner Downing St and more discursive Cabinet meetings. In 1997, Tony Blair quickly expanded the size and scope of Downing St operations, including in foreign affairs, but Cabinet and Cabinet committees began again to play a smaller role in decision making.

In June 2001 Blair placed Cabinet Office secretariats behind his Europe and Foreign Affairs Advisers. After the terrorist attacks of 11 September 2001, Blair also upgraded the long-standing Intelligence Coordinator post to incorporate security within its portfolio.

These changes meant a significant increase in capacity at the centre. Blair’s first Europe Adviser, Sir Stephen Wall, felt that this actually made No.10 too powerful in foreign policy, upsetting the delicate balance between the centre and the Foreign Office. Questions of central co-ordination and the overlap between intelligence and policy were central to the Iraq War criticisms.
As prime minister, Gordon Brown continued Blair’s practice of employing three senior officials as advisers on Europe, Foreign Policy and Security, but David Cameron’s NSC reforms combined these advisory functions in one new post, the National Security Adviser, supported by a well-resourced national security secretariat.

The NSC therefore bears strong resemblance to previous iterations of national security co-ordination at the centre of government. It continues a post-9/11 trend under Blair and Brown that focused on ‘security’ as a more salient co-ordinating factor than other aspects of defence and foreign policy. Yet it still deserves to be regarded as new in marking a significant further step in both streamlining and upgrading central support for the prime minister. The NSC has also met more frequently and benefited from more sustained prime ministerial attention than its recent predecessors.

It would be wrong to imagine that there is one, timeless and perfect way of co-ordinating national security from the centre. Like any other area, national security co-ordination depends greatly on a prime minister’s personal style and approach. As we consider the future of the national security machinery we should examine how well it is currently operating. But it is also worthwhile to ponder how it has been shaped by – and whether it can learn from – its own history.

Sausages, Evidence and the Messiness of Policymaking

Professor Jonathan Grant and Dr Benedict Wilkinson

“Law and sausages”, as Otto von Bismarck famously did not say, “are two things you do not want to see being made”. Much the same might be said of policymaking. Like sausage- and law-making, it is messy, iterative, and little understood. And for those of us who champion the use of evidence in policymaking – which is the central theme that runs through this month’s PolicyWonkers blog, and, more broadly, the work of the Policy Institute at King’s – the messiness inherent to all policymaking is a real challenge.

Indeed, as we see it, part of the appeal of evidence is that it ‘tidies up’ some of the messiness that pervades the policymaking process. To apply the words of David Sackett, one of the pioneers of evidence-based medicine, to a slightly different context: good policymaking should be about “the conscientious, explicit and judicious use of current best evidence”.

Indeed, this was precisely the idea captured by New Labour in their 1997 Manifesto when they portrayed themselves as “a party of ideas and ideals but not of outdated ideology. What counts is what works”. Nor is this a party political issue: David Cameron, for example, recently introduced the Contestable Policy Fund to “commission high quality advice from outside the Civil Service [and] draw directly on the thinking, evidence and insight of external experts”.

The use of evidence in policymaking is not just happening in the UK. In Canada, probably the second most vociferous advocate for the utility of evidence in policy formulation after the UK, for example, the Canadian Health Services Research Foundation has received federal funding to “provide healthcare policy- and decision-makers with the robust, accessible research they needed to make evidence-informed improvements to healthcare financing, management and delivery”. And although the EU has been somewhat reluctant to extol the virtues of evidence in policymaking, Anne Glover, the EU’s Chief Scientific Advisor, recently called for “a new system of evidence gathering within the Commission that entirely disconnects evidence gathering with the political imperative” and advocated for the creation of a department in the EU whose sole purpose is to examine policy proposals against the best available evidence*.

The growing acknowledgment of – and advocacy for – the use of evidence in policymaking is, in our view, a good thing. But it is balanced by a certain cynicism about what government can actually achieve. A recent poll conducted by Populus on behalf of the Institute for Government, for instance, concluded that “two-thirds of the public say they would be more likely to vote for a party that demonstrates how it would implement its manifesto pledges… But few (only 15%) are confident that parties know how they will fulfil their promises in government.” The fatalistic view, whether from government or the voters, that ‘nothing works’ is dangerous and one that needs to be challenged. Champions of “the conscientious, explicit and judicious use” of evidence in policymaking need to up our game.

For us, this means two things. First we need to understand and acknowledge that ‘evidence’ is just one ingredient in the policymaking process: habit, resources, values, ‘marketability’ and politics all, for example, legitimately impinge on that way policy is made – and, arguably, should be made. From this perspective, one role that evidence can and does play is to help reduce the uncertainty – the messiness – inherent to policy and decision-making.

But that is not to say this is the only role that evidence can play. And this is our second point: we need to accept that there has been a failure to bridge the so-called ‘gap’ between research and policy. In our view, this needs to be combatted by adopting new paradigms for engagement and exploring innovative ways of working. A recent systematic review of the barriers to – and facilitators of – the use of evidence by policymakers suggests that little has changed over the past decade: timely access to good quality and relevant research evidence, collaborations between researchers and policymakers, relationship and skill-building with policymakers – these are the factors that continue to be the most important in influencing the use of evidence.

But bearing in mind the still relatively meagre role played by evidence in the process, the fact that little has changed suggests that our current models are broken. We need to explore new ways of engaging with policymakers: developing communities of practice, employing the concept of ‘nudging’ from behavioural economics, and, most importantly, creating a ‘marketplace’ for evidence that includes demand-side incentives.

At the Policy Institute at King’s we have been reflecting on ‘what works’ in maximizing the impact of academic research on policy and practice.  Learning from the research on evidence-based policymaking we have recently refreshed our strategy where we have organised ourselves around three themed activities: mobilizing impact; building partnerships and delivering analysis. Our mission is to improve evidence-informed policy by facilitating mutual engagement between academic, business and policy communities around current and future policy needs. This blog is one part of that strategy: each month, we will invite four individuals to write short, ‘chatty’ posts that communicate their evidence-based policy ideas across a broad range of topics – from social care to Europe.  We very much hope that you have enjoyed this post – and will continue to read the blog in the future.

*Three days after this post was published, the European Commission announced it was scrapping the post of Chief Scientific Advisor; for champions of evidence-based policy, as we are, this is a great disappointment, and we hope that this is not part of a wider and negative attitude towards the role of evidence in policy in the EU.