What outcome measures are collected, how do you use them and how do they demonstrate improvement?
Personality Disorder Managed Clinical Network
Since its inception the PDMCN has focused on the collection of clinical outcomes and service user satisfaction data to understand the effectiveness (or not) of our interventions, to monitor treatment progress and to obtain specific feedback about aspects of service user experience. There is a well-established system for the administration, scoring, analysing and reporting on these areas, which results in an Annual Review Report. The report for 2018/19 will be completed by July 2019. Service user involvement is evident throughout the whole process, culminating in co-produced recommendations.
A number of PROMs, which have been agreed with service users, are used. These include:
- World Health Organisation Quality of Life measure (WHOQoL)
- Clinical Outcomes in Routine Evaluation (CORE-OM)
- The Borderline Evaluation of Severity over Time (BEST)
- Occupational Self-Assessment (OSA)
In our 2017/18 Annual Review (see Appendix 2), service users within the PDMCN showed modest improvements in terms of psychological distress, interpersonal functioning, and quality of life. 84% of service users felt positively about the Network and the care and treatment received.
Family and Carers
The service delivers six week cohorts of carers’ groups three times per year and at different times of the day including evenings. The group aims to provide education about personality disorder, to support carers in thinking about what caring for someone with a personality disorder might entail, and to consider self-care as an essential part of caring. The first groups in 2016 were co-facilitated by a clinical psychologist and care co-ordinator; groups are now co-facilitated with a Carer Consultant. Outcome measures used include:
- The Personality Disorder: Knowledge and Skills Questionnaire (PD-KASQ) adapted specifically for this group. The PD-KASQ measures understanding, capability and emotional responses regarding working with people with a personality disorder.
- The Short Warwick-Edinburgh Mental Well-being Scale (SWEMWBS) which is a brief self-report measure of wellbeing.
- The Carer Satisfaction Questionnaire (Larsen et al.) adapted for this group.
Outcomes suggest that carer knowledge and understanding of personality disorder significantly improves as a result of the group, as does carer wellbeing. Carers also report high overall satisfaction with the group with all responses falling into the ‘very satisfied’ or ‘mostly satisfied’ categories.
Pathway Development Service
The PDS collects routine evaluation data from key stakeholders in order to evaluate the clinical impact of PDS Reviews. Questionnaires are completed by Case managers, Clinical Team members and Service Users following Reviews and Housing and Resettlement Brokerage. This method of evaluation has been ongoing since 2016 and has led to important changes in service delivery, for example:
- Improving the clarity of reports and recommendations: A Glossary of Terms have been developed in conjunction with experts by experience which is now included as an appendix to reports.
- Improving access: PDS letters have been reviewed to ensure that the referral and review process is clear. A new service information leaflet for service users and carers has been developed in conjunction with experts by experience to ensure that clear and accessible information is available and a short animated film is being co-produced with service users.
- Improving Response Rate: The online tool ‘Survey Monkey’ was introduced in September 2017 to provide easier access to the Questionnaires. In addition, recommendations from Market Research have been followed to improve email and written invitations to take part in the evaluation.
- Improving clarity of questionnaires for service users: Further consultation with service users has led to changes in wording of questions in the Service User Questionnaire to ensure clarity of meaning.
- Increasing contact between PDS caseworkers and wards: PDS staff have arranged time to shadow staff within inpatient units in order to improve understanding of current issues in inpatient services. Similarly, Housing and Resettlement workers have arranged to shadow local ‘wrap around’ teams or other appropriate accommodation services.
Responses from stakeholders are broadly positive, with respondents describing PDS interventions as helpful in supporting team thinking about the needs of service users and in planning suitable care and resettlement pathways. Commissioners and case managers report feeling very satisfied with the service, as do the majority of clinical staff and service users.
Yorkshire/Humberside Personality Disorder Partnership (Offender Personality Disorder Services)
Routine outcome measurement for our IIRM service aims to address outcomes as outlined in the National IIRMS specification. These include: prevention of reoffending and increasing offenders’ awareness of their impact on their victims; harm reduction; recovery; improving wellbeing; rehabilitation and social inclusion.
Global questionnaires measuring engagement, wellbeing and protective factors include measures completed by probation officers (Treatment Engagement Rating Scale (TER) (Drieschner & Boomsma, 2008) alongside self report measures to be completed by participants:
- Clinical Outcomes in Routine Evaluation (CORE-OM)
- Reflective Functioning Questionnaire (RFQ)
- Structured Assessment of PROtective Factors for violence risk (SAPROF)
- Occupational Self Assessment (OSA) (v2.2)
- Patient Health Questionnaire (PHQ-9)
- Generalised Anxiety Disorder Assessment (GAD-7)
- Overt Aggression Scale (OAS-M)
The IIRMs teams also complete outcome measures regarding each participant:
- Treatment Engagement Rating Scale (TER)
This list is not exhaustive and bespoke, individualised and goal orientated measures may also be
We have learned that it is problematic gathering outcome data for YHPDP and small sample sizes make it impossible for the service to draw any firm conclusions about impact using these measures. The service is, therefore, concerned with co-producing a narrative about what is meaningful for service users so that we can better ensure that we are utilising measures which register real clinical impact.