Comorbidities of dementia in electronic health records: validation of a natural language processing app and comparison with data linkage

Awareness of comorbidities in people with dementia is vital context for dementia research and clinical management. Natural Language Processing (NLP) applied to free text in the electronic health record (EHR) may help ascertain comorbidity presence. Our aim was to assess the validity of the MedCAT Comorbidity NLP app in a cohort of people with dementia. We used text from the EHR of South London and Maudsley NHS Mental Health Trust via Clinical Records Interactive Search (CRIS), linked to structured data from local primary care (Lambeth DataNet) and hospital admission data in a cohort of 3,543 people with dementia. We evaluated MedCAT Comorbidity, examining internal validity via manual review of text and external validity by cross-validation against linked primary care and hospital data; and comparison of the predictive value of comorbidity measures for mortality. Internal precision was highest for atrial fibrillation (100%) and lowest for diabetes (78%). Cross-validation with linked data varied substantially across conditions, with precision from 96% to 59% and recall 77% to 4%. The median number of comorbidities identified was two using any single source, or three when combined. MedCAT Comorbidity demonstrated predictive ability for mortality (HR 1.11 per condition), although the model fit was better using comorbidity from hospital admission data. In people with dementia, the MedCAT Comorbidity app showed high precision for most conditions during internal validation. The variability of precision and recall on external validity assessment partly reflects incomplete documentation. Results from the MedCAT comorbidity app can assist researchers in understanding comorbidity of cohorts, particularly when primary care or hospital data is not available. Our findings highlight the need to develop quality assurance protocols that include internal and external validity checks before deploying general data extraction apps for specific clinical subpopulations.