Large-scale evaluation of an AI system as an independent reader for double reading in breast cancer screening

ImportanceScreening mammography with two human readers increases cancer detection and lowers recall rates, but workforce shortages make double reading unsustainable in many countries. Artificial intelligence (AI) as an independent reader in double reading may support screening performance while improving cost-effectiveness. The clinical validation of AI requires large-scale, multi-vendor studies on unenriched cohorts. ObjectiveTo evaluate the performance of the Mia(R) AI system on data that the AI system would process in real-world deployments. DesignA retrospective study simulating the impact of AI on an unenriched screening sample. SettingSeven European breast screening sites representing four centers: three from the UK and one in Hungary (HU), between 2009 and 2019. ParticipantsThe sample included 275,900 cases (177,882 participants) from seven screening sites, involving two countries and four hardware vendors from 2009 to 2019. InterventionSimulation of double reading using AI as an independent reader in breast cancer screening on historical data. Main Outcomes and MeasuresPerformance was determined for standalone AI compared to the historical single reader and for simulated double reading with AI compared to historical double reading, assessing non-inferiority and superiority on relevant screening metrics using a non-inferiority margin of 10% relative difference and a one-sided alpha of 2.5% for both tests. ResultsStandalone AI detected 29.8% of missed interval cancers. When compared with historical double reading, double reading with AI showed non-inferiority for sensitivity and superiority for recall rate, specificity and positive predictive value. AI as an independent reader reduced the workload for the second human reader but increased the arbitration rate from 3.3% to 12.3%. Applying the AI system could have reduced the human reading time required by up to 44.8% and reduced the recall rate by a relative 7.7% (from 5.2% to 4.8%). Conclusions and RelevanceUsing the AI system as an independent reader maintains or improves the double reading standard of care, while substantially reducing the workload. Thus, it has the potential to provide operational and economic benefits. Trial RegistrationRegistered on ISRCTN, study ID: ISRCTN18056078