Managing very small scale objective structured clinical examinations - can we ensure reliability of pass/fail decisions with low candidate numbers? A feasibility study of global score only decision making

BackgroundThis study assesses a change to pass/fail decision making from an Angoff method to one based on examiner global judgements in very small scale Objective Structured Clinical Examinations (OSCE). ApproachData was collected from a Physician Associate OSCE in which two examiners marked each station one utilising a global rating based mark scheme and the other the original, Angoff based, mark scheme. Alternative global ratings were used to try and improve the judgement decision of examiners and checklists significantly reduced into a small number of broader domains. Examiners were also asked to provide some feedback on the new rating scales. EvaluationMann-Whitney U tests were used to evaluate the data. Overall there was fairly good correlation of candidate performance between the two methods. Some anomalies were found with the use of the word safe in the global ratings creating a "killer" station. Examiners had mixed views but most were positive about the change in approach. ImplicationsThe initial results seem promising suggesting that global rating scales alone may be suitable to determine pass/fail decisions in very small scale OSCE with implications for all educators managing such assessments. We intend to keep the adjusted global rating scales.