Dr Mary Baginsky is Senior Research Fellow at the NIHR Policy Research Unit in Health and Social Care Workforce. (1,109 words)

The report on the second round of Children’s Social Care Innovation Programme funding for the 10 MTM (Munroe, Turnell and Murphy) Signs of Safety (SofS) pilots has just been published. The strengths-based approach to child protection and safety is widely used around the world, as well as in two-thirds of local authorities in England (Baginsky et al., 2020). The term ‘Signs of Safety’ refers to a model of practice that consists of:

• principles that privilege relationships with children and their families
• disciplines in relation to assessments, behaviours and language
• tools for assessment and planning, as well as for use with children and families.[1]

It is not known to what extent a ‘pure model’ of SofS is in place in English local authorities. The survey that identified its use also showed the variations – and pick and mix approaches – that  were in place.

Back in 2014 when the 10 MTM pilots were recruited the idea was that they worked to the model above which was developed into a whole system design that MTM considered essential to support, monitor and build high-quality SofS practice based on a supportive organisational culture and the commitment of those in senior leadership positions. The first evaluation found that:

• in some areas engagement with families had improved
• where SofS was more embedded a higher proportion of families reported that their goals had been achieved
• SofS had the potential to improve services for children and their families, but there was no evidence of improved outcomes.

The absence of comparison sites and the fact that the initiative had only been in place for a short time meant it was not possible to link changes over time and causally.

When MTM was given a second round of funding to support the pilots an evaluation was also commissioned. By the second evaluation one of the pilots had dropped out after receiving a judgement of inadequate from Ofsted and was replaced by another authority that had been using SofS for some time and had moved from a ‘requires improvement’ judgement in 2014 to one of ‘outstanding’ in 2018. Two of the 10 pilots were also judged to be ‘inadequate’ during the evaluation and one left the project. It was possible, however, to explore the reasons why the two pilots did not continue with SofS and these are reported.

Unfortunately in the second evaluation it was not possible to conduct the work across all ten pilots or to engage with children and their parents in similar ways to the first study. Funding was only available to look in depth at five of the 10 as case studies, but we were able to conduct a contrast study to compare two pilot authorities and two authorities that did not use SofS at all but had their own practice frameworks in place – one using ‘restorative practice’ and the other ‘strengthening families’. Although this was a limited study it responded to one of the recommendations in the report on the first evaluation and provided the opportunity to accompany social workers visiting families and observe their practice. In addition to examining family and child outcomes through the contrast study and case record analysis, they were also explored using national data sets and by a Difference in Difference (DiD) analysis undertaken by What Works for Children’s Social Care (WWCSC), using individual-level data from the National Pupil Database. A cost benefit study was also conducted using costs collected from the pilots and applied to outcome data from national datasets to explore any savings from changes in outcome via unit cost data.

We found evidence that the pilots had taken steps to improve the implementation by, amongst other things, aligning their recording systems with SofS, conducting more focused audits and accessing training that was available from MTM. However there was considerable variation across the pilots in the consistency with which the elements of SofS were used and observed. The interviews also identified a surprising lack of consistency about what SofS is while, at the same time, it appeared that Elia – the organisational home of SofS – was being more prescriptive about its implementation.  One of the messages given to pilots during the project was that the increasing bureaucratisation of social work was damaging, but during this second evaluation the requirements around recording from MTM led to those in the case study pilots observing that the level of prescription had increased rather than decreased during this time.

There was also a lack of evidence from local authority level date and the DiD analysis of improved outcomes for children. The reason for this may or may not  lie in the findings of the contrast study. Observation of practice found no differences in practice between the sites using SofS and those not using it. Evidence from the measures that were used – Yatchmenoff Client Engagement Scale and the Working Alliance Inventory – showed no significant differences between parental involvement in pilot and contrast sites. On the basis of  these observations more collaborative relationships between social worker and parent were linked with overall clinical and professional competence, measured by Clinical Competence-Based Behavioural Checklist (CCBC) (Lu et al., 2011), than the use of any one approach. The extent to which SofS was used by social workers had no significant impact on this relationship, although in many instances it had been difficult to identify its use in interactions with families and there were indications that restorative practice –  as used in one the contrast sites – had more impact on the quality of social workers’ interactions with families.

It is the contrast study that offers the most promise as far as any future evaluation is concerned. In the past a few randomised controlled trials (RCTs) have been conducted involving SofS, but they have involved small numbers and not included details of the intensity of implementation. While it is widely assumed that RCTs provide the strongest evidence for causality and for effectiveness they require a consistency of intervention which is not yet observable in SofS. What we have produced is a template for a contrast study and the experience of this evaluation points the way for a quasi-experimental study that would advance our knowledge of SofS specifically, and interactions with families in general, which is much needed.

Mary Baginsky is Senior Research Fellow at the NIHR Policy Research Unit in Health and Social Care Workforce.

Report and appendices, published 28 January 2021

Baginsky M, Hickman B, Harris J, Manthorpe J, Sanders M, O’Higgins A, Schoenwald E and Clayton V. (2021) Evaluation of MTM’s Signs of Safety Pilots. Evaluation report. London: Department for Education.

Baginsky M, Hickman B, Harris J, Manthorpe J, Sanders M, O’Higgins A, Schoenwald E and Clayton V. (2021) Evaluation of MTM’s Signs of Safety Pilots. Evaluation report appendices. London: Department for Education.

References

Baginsky, M., Ixer, G. and Manthorpe, J. (2020) Practice Frameworks in Children’s Services in England: An Attempt to Steer Social Work Back on Course? Practice, doi:10.1080/09503153.2019.1709634

Lu, Y. E., Ain, E., Chamorro, C., Chang, C., Feng, J. Y., Fong, R., and Yu, M. (2011). A new methodology for assessing social work practice: The adaptation of the objective structured clinical evaluation (SW-OSCE). Social Work Education, 30(2), 170-185. doi:10.1080/02615479.2011.540385

Yatchmenoff, D.K. (2005) Measuring client engagement from the client’s perspective in nonvoluntary child protective services. Research on Social Work Practice, 15, 2, 84–96.

[1] https://www.signsofsafety.net/