Development and implementation of an AI system for clinical toxicology sign-outs

BackgroundModern natural language tools have potential to improve clinical workflows, but few have been successfully deployed in practice. Here, we present the development, deployment, and evaluation of an AI language tool for generating preliminary clinical sign-outs in a urine drug testing service. MethodsLarge language models (LLMs) were used to extract substance use patterns from 83,553 urine drug test interpretations. We then trained an AI model using these data to predict substance use from qualitative and quantitative urine testing results. Predicted substance use patterns were used to create preliminary clinical sign-out statements, which were then integrated into an existing clinical workflow. Pre- and post-deployment user studies were performed to evaluate model performance and user experience within this workflow. ResultsLLM-based extraction of substance-use patterns was 99.9% accurate, outperforming human labelling. Substance use prediction was similarly accurate, with area under the ROC curve > 0.99 across 33 drug categories. Workflow integration reduced clinical sign-out times by 65s per case (51% efficiency gain), with the greatest benefits seen for less experienced users. ConclusionsAI-based interpretation of urine drug testing was fast and accurate, providing significant efficiency gains to the clinical service. This demonstrates that natural language tool integration can provide substantial clinical benefit, without comprising quality of care.