Same Inputs, Different EDSS: Measuring Specification Drift in Clinical Scoring Pipelines
Hwang, S.; Mowery, D. L.; Thomas, S.; Williams, H.; Bar-Or, A.; Sharma, V.; Buijs, F.; Perrone, C.
Show abstract
Clinical informatics pipelines increasingly compute validated clinical endpoints from upstream NLP outputs. Even when the endpoint is defined by an established rubric, translating that rubric across representations - natural language instructions, program logic, and reference implementations - can introduce specification drift, where ostensibly equivalent calculators yield meaningfully different scores. We study this phenomenon for the Expanded Disability Status Scale (EDSS), a standard measure of disability in multiple sclerosis. Holding constant a shared set of functional system (FS) subscores extracted by a large language model (LLM), we compare EDSS values computed across three representations of the same scoring rubric: prompt-executed natural language, LLM-generated code, and a canonical reference implementation. We characterize disagreement structure, distributional shifts, and clinically salient boundary flips, and we propose an audit workflow that treats endpoint computation as a first-class verification target in clinical NLP systems.
Matching journals
The top 6 journals account for 50% of the predicted probability mass.