Retrospective Evaluation of an AI System to Classify Negative Musculoskeletal Trauma Radiographs

Study objectivesTo evaluate an AI system for musculoskeletal (MSK) radiography in confidently identifying examinations without injury-related pathologies to support AI-guided discharge of patients without traumatic findings. MethodsWe retrospectively sampled radiographic examinations, including one or more radiographs and a radiological report, of suspected MSK trauma from >1,000 clinical sites in two countries. Medically trained professionals independently classified all exams. When disagreements between the classification and the original radiological report arose, adjudication was performed by a third professional. Annotators were also asked to record their confidence level of each classification. The AI system analyzed all exams and assigned them to the categories AI Positive, AI Negative, and AI Negative (very high confidence). Performance of the AI system was assessed using error rate, false negative rate (FNR), and qualitative review of misclassified exams. ResultsA total of 2,962 exams were included. The AI classified 27.6% (818/2,961; 95% CI: 26.0-29.3) of exams as AI Negative (very high confidence). Of all exams, 0.7% (21/2,962; 95% CI: 0.0-1.0) were falsely classified as highconfidence negatives, corresponding to a false negative rate (FNR) of 2.0% (21/1,026; 95% CI: 0.0-2.9). Qualitative review of false negatives showed that the majority had no clinical consequence if correctly diagnosed during routine follow-up the following day, and no clearly high-risk exams were missed. ConclusionThe AI system identified over one-quarter of MSK trauma radiographs as confidently negative with a very low rate of false negatives, performing on par or better than the reported standard-of-care. These results suggest potential for safe, AI-driven decision support and workflow optimization for the discharge of patients with clearly negative examinations.