A methodology for selection and quality control of the radiological computer vision deployment at the megalopolis scale

In recent years, there has been tremendous interest in the use of artificial intelligence (AI) in radiology in order to automate the interpretation. However, uncontrolled and widespread use of AI solutions may have negative consequences. Therefore, before implementing such technologies in healthcare, thorough training of personnel, adaptation of information systems, and standardized datasets for an external validation are required. All this necessitates a formation of a unique unified methodology. The best practices of AI introduction in diagnostic radiology are still subject to debate and require new results of a scientific-practical research with the assessment of implementation conditions. This work discusses expected issues and potential solutions for the introduction of computer vision-based technologies for automatic analysis of radiological examinations with an emphasis on the real-life experience gained during simultaneous AI implementation into practice of more than a hundred state radiology departments in 2020-2021 in Moscow, Russia (an experiment). The experiment used end-user software testing approaches, quality assurance of AI-based radiological solutions, and accuracy assessment of the AI-empowered diagnostic tools on local data. The methods were adapted and optimized to ensure a successful real-life radiological AI deployment on the extraordinary large scale. The experiment involved in total around thousand diagnostic devices and thousand radiologists. AI deployment was associated with additional options in a routine radiologists workflow: triage; additional series formed by AI with indication of pathological findings and their classification; report template prepared by AI in accordance with the target clinical task, user feedback on AI performance. A multi-stage methodology for implementing AI into radiological practice that was developed and advanced during the experiment is described in this report. EssentialsO_LIA methodology for the AI deployment for non-academic radiological sites excluded more than half of the offered AI solutions that do not fulfill the diagnostic and functional requirements C_LIO_LIQuality control of AI should be supported by not only data scientists, IT specialists or engineers, but also by radiologists at all stages of selection and testing. C_LIO_LIRadiologists need to understand the capabilities, limitations of AI by getting an additional training. C_LI