Reliability testing of patient-reported measurement instruments of Shared Decision-Making within surgical treatment pathways: a mixed-methods study

ObjectivesTo evaluate the test-retest reliability of two instruments measuring shared decision-making (SDM) and to explore factors affecting the stability of participants lived experience of SDM during the test-retest interval. DesignMixed-methods prospective cohort study, nested within an ongoing National Health Service (NHS, United Kingdom) Trust Quality Improvement study SettingSeven surgical specialties within a tertiary NHS Hospital Trust. Participants/ProceduresPatients (>18 years) booked for elective surgical interventions were invited to complete two measurement instruments at each timepoint: CollaboRATE (a 3-item questionnaire developed to assess patients perception of professionals SDM skills) and SHARED (a 10-item questionnaire to assess patients experience of making a treatment decision with health professionals). Instruments were completed at baseline timepoint (date of surgery booking consultation) and 5-10 days after consultation, but prior to surgery (retest timepoint). Main outcome measuresIntra-class correlation (two-way, mixed, absolute agreement) and Kappa coefficients at item, total- and top-score levels were calculated. Bland-Altman plots were used to describe agreement between initial and retest measurement for each instrument. Semi-structured interviews to explore participants lived experience were conducted remotely after retest measurement with a purposively selected sample of patients of varying socio-demographic characteristics, surgical specialties, and direction of score change. Transcriptions underwent thematic analysis using inductive coding approaches to identify themes. Results86 patients completed retest measurements (median time to completion = 8 days). Test-retest reliability was weak for CollaboRATE (ICC=0.34, p<0.001) and moderate for SHARED (ICC=0.52, p<0.001) total scores. Test-retest reliability was moderate for CollaboRATE top score ({kappa} = 0.47, p<0.001). Interviews with nine patients identified two key themes driving instability in the test-retest interval: 1) ongoing reflection on the SDM process, and 2) a need for more support for SDM. ConclusionOur study demonstrated weak-moderate reliability in measuring patient-reported SDM which may be explained by patients continued reflection on decisions after surgical consultations. Future research should consider the fact that SDM is a process, and work is needed to understand how and when to optimally measure SDM so that the impact of continued reflection and reasoning are not missed. KEY MESSAGESO_ST_ABSWhat is already known on this topic?C_ST_ABSO_LIInstruments (e.g. self-reported questionnaires) measuring SDM from the patient perspective are widely used in clinical pathways, in research, and to guide the development of healthcare services. Few studies have evaluated the test-retest reliability of these instruments, and those done to date report poor reliability. C_LI What this study adds?O_LIThis mixed-methods study explored the test-retest reliability of two SDM measurement instruments in the context of decision-making around surgical interventions. It is the first to supplement standard test-retest methods with interviews to explore patients perceptions of SDM over time in this setting. We found that patients reflect on decision-making processes and consider how this may affect the interpretation of reliability metrics in the context of SDM measurement. C_LIO_LIOur results affirm the dynamic nature of SDM and suggest that test-retest reliability may therefore not be an indicator of the overall quality of an SDM measure when used in this context. C_LI How this study might affect research, practice or policy - summarise the implications of this studyO_LISDM measurement instruments have different levels of reliability. Routine SDM measurement may identify a subgroup of patients who reflectively question their decisions for medical treatment in the time following consultation. There is scope to target interventions to improve SDM in the period between consultation and procedure. Clinicians and policymakers should design pathways with the timing of SDM measurement kept in mind. C_LI