Performance and change impact analysis of a commercial artificial intelligence tool for radiographic knee osteoarthritis grading and joint space width measuring

1.Background and rationaleKnee osteoarthritis (OA) is a common disease characterized by reduced function, stiffness, and pain. This clinical diagnosis is commonly supported with radiography of the weight-bearing knee. Radiographic features, such as the Kellgren-Lawrence (KL) grading system, are used as eligibility criteria for clinical studies while others, such as the OARSI grades and minimal joint space width, are used as endpoints for structural OA progression. A higher preoperative KL-grade has been correlated with better pain- and functional outcomes after knee arthroplasty surgery. Consequently, the KL-grade is a common requirement for approving knee arthroplasty among health insurance providers and it is commonly used by orthopedic surgeons as part of determining knee arthroplasty candidacy. Historically, a radiologist was required to draw on and grade radiographs of the knee to extract the features. With increasing computational power and the increased use of deep convolutional neural networks, off-the-shelf artificial intelligence (AI) tools have become available for automatic extraction of these features. They have received regulatory approval for commercialization, but it is apparent that more diligent external validation is required. Finally, as AI tools begin to mature, new versions are released. It is important to assess how these developments change the current performance of the tool. ObjectivesThe aim of this analysis is to evaluate the performance of a commercially available AI tool and of readers with different experience levels in orthopedic surgery and radiology at clinically relevant Kellgren-Lawrence grading system thresholds. Additionally, the performance of the AI tool for OARSI grades and patellar osteophytes will be evaluated across two versions of the AI tool. MethodsThis study is a secondary analysis of the data from the AutoRayValid-RBknee study, a retrospective observer performance study. It consists of non-fixed-flexion radiographs acquired from the production picture archiving and communications system (PACS) from three European centers. The primary outcome will be the difference in area under the receiver operating curve (AUC) between the readers and the AI tool at the prior authorization clinical criteria threshold (KL [≥] 3). Key secondary outcomes will be radiographic knee osteoarthritis (KL [≥] 2), osteoarthritis clinical trial inclusion (2 [≤] KL [≤] 3), and weight-loss trial inclusion (1 [≤] KL [≤] 3). The AUC of the readers will be computed using the SROC approach as proposed by Oakden-Rayner et al. Further, the performance of the AI tool for grading ordinal OARSI grades will be evaluated using the ordinal ROC as proposed by Obuchowski et al. and the AUC is used for estimating binary OARSI-grade and patellar osteophyte classification performance. PopulationPatients with knee pain referred for radiography on suspicion of knee osteoarthritis. Index testO_ST_ABSReadersC_ST_ABSEach center will recruit four readers from across radiology and orthopedic surgery, one in-training and one board-certified for each specialty. AI toolRBknee-2.2.0 (CE version, KL-grading, OARSI grading, patellar osteophytes) and RBknee-2.1.0 (CE version, KL-grading, OARSI grading, patellar osteophytes) will be used to perform the change impact analysis of advancing product development. Reference testThe reference standard will be determined by the majority vote of three readers, one from each participating hospital who are a board-certified musculoskeletal radiology consultant with expertise in clinical and research evaluation of KOA including extensive experience using the KL-grade. Further statistical detailsO_ST_ABSSample sizeC_ST_ABSNot applicable as this is a secondary analysis. FrameworkThis is a diagnostic test accuracy study assessing the performance of a commercially available AI tool for radiographic evaluation of knee osteoarthritis according to established grading systems. Additionally, change impact analysis will be performed where multiple versions of the AI tool are available. Confidence intervals and P valuesAll 95% confidence intervals and P values will use an alpha of 5%. MultiplicityNo explicit multiplicity correction will be performed. Instead, a hierarchical approach will be taken based on tabular order of the tested hypotheses in Table 3. O_TBL View this table: org.highwire.dtl.DTLVardef@5ca41corg.highwire.dtl.DTLVardef@1a38d0org.highwire.dtl.DTLVardef@187e2d1org.highwire.dtl.DTLVardef@3c1990org.highwire.dtl.DTLVardef@1b254a6_HPS_FORMAT_FIGEXP M_TBL O_FLOATNOTable 3.C_FLOATNO O_TABLECAPTIONComparison of the AI tool and the readers for dichotomized Kellgren-Lawrence grading at prespecified thresholds Values will be reported as the AUC (95% confidence interval). Unassisted readers will refer to the readers without assistance from the AI tool and AI assisted readers to when the readers did receive AI assistance. OA, Osteoarthritis; OARSI, Osteoarthritis Research Society International; AUC, area under the receiver operating curve; SROC, summary ROC; ROC, receiver operating curve. C_TABLECAPTION C_TBL Statistical softwareR version 4.2.2 (or newer).