A comprehensive evaluation methodology for the publicly accessible AI services for medical diagnostics

Online AI in telemedicine for radiology became widely available and valuable in the pandemic, particularly for chest CT analysis. On the other hand, the potentially harmful consequences of such services inappropriate usage cannot be neglected. Thus, a suitable methodology for quality assurance and quality control has to be established. A studys purpose was to develop and test an original methodology for a complex evaluation of open-access AI services in teleradiology. The approach included assessing the user experience, accessibility, safety, and diagnostic accuracy on the independent reference dataset. The methodology was applied to assess seven AI services for the detection of COVID-19 on a CT scan. A comparative analysis of this assessment is presented in this work. The analysis allowed us to draw conclusions about AI services quality and their value for different users - patients, physicians, and healthcare data scientists. The originality of the findings, timeliness, and interdisciplinary approach make this quality assurance methodology of particular interest for further application and spreading. HighlightsO_LIThe availability of diagnostic procedures increases each year with a corresponding increase of low-impact workload. C_LIO_LIDuring pandemics, medical AI services have become valuable in reducing the workload on healthcare professionals. C_LIO_LIQuality assurance for AI in healthcare requires an interdisciplinary approach - medicine, IT, and data science collaboration. C_LIO_LISuggested methodology of AI quality assurance suits different target users groups, such as the general public, physicians, and data scientists. C_LI