A systematic comparison of tools for predicting antimicrobial resistance from nanopore sequence data

Antimicrobial resistance (AMR) presents a pressing need to ensure that the right antimicrobials are used to target the right microbes at the right time. Ideally, the appropriate antimicrobial is selected after patient samples have been cultured and assessed with antimicrobial sensitivity testing (AST). However, the time needed for culture-based diagnosis leads to immediate empirical treatment, often with broad-spectrum and/or high-tier antimicrobials. Direct nanopore metagenomic whole genome sequencing to identify pathogens and predict their antimicrobial resistance is a rapid and patient-side alternative. A limitation of this approach is potential inconsistencies in in silico predicted AMR phenotypes. Here, we benchmarked the current performance of in silico AMR prediction strategies for nanopore-generated long read data. Using nanopore data paired with AST phenotyping for 201 samples, we assessed the impact of basecalling mode, data volume, and assembly strategy, and compared the performance of eight in silico AMR prediction tools with seven AMR databases. We found that basecalling accuracy mode does not affect the overall accuracy of in silico AMR predictions, but assembly strategy and data volume both do. Prediction tools using the ResFinder database scored best for balanced accuracy (0.80 {+/-} 0.02 for both ResFinder and ABRicate), whilst DeepARG scored best for sensitivity (0.65 {+/-} 0.03). However, even the best performing in silico AMR prediction strategy missed some resistance identified by lab-based AST. In silico AMR prediction can therefore supplement lab-based AST, but cannot yet replace it. Impact statementAntimicrobial resistance (AMR) is threatening modern standards of human and veterinary healthcare. Rapid and patient-side diagnostic tests are needed to diagnose bacterial infections and allow clinicians to select effective antibiotics. Current tests based on bacterial cultures take several days, which may delay diagnosis and treatment, or lead to inappropriate "just in case" treatment while waiting for the results. In contrast, nanopore metagenomic whole genome sequencing can identify bacterial infections and predict which antibiotics will be effective in minutes to hours. However, the accuracy of these tests is uncertain. We therefore compared the performance of eight AMR prediction tools and seven databases of AMR determinants, using 201 bacterial samples with known antibiotic susceptibility and resistance. We found that the sensitivity (i.e. false negative rate), specificity (i.e. false positive rare) and overall accuracy of the tools and databases varied. In particular, even the best performing AMR prediction methods missed some AMR. Therefore, while these tools are useful for rapid and patient-side diagnosis and treatment decisions, they still have limitations and should be used alongside bacterial cultures and antibiotic sensitivity testing. Data summarySequencing data for the samples sequenced for this study are available at the NCBI under BioProject ID PRJNA1292816 (SRA accessions for all datasets used here are available in Supplementary Table S1). All commands and code used can be found at: https://github.com/nataliering/nanopore_AMR_tools/ The authors confirm all supporting data, code and protocols have been provided within the article or through supplementary data files.