Development and evaluation of an interoperable natural language processing system for identifying pneumonia across clinical settings of care

ObjectiveTo evaluate the feasibility, accuracy, and interoperability of a natural language processing (NLP) system which extracts diagnostic assertions of pneumonia in different clinical notes and institutions. Materials and MethodsAn NLP system was designed to identify assertions of pneumonia in three types of clinical notes from electronic health records (EHRs): emergency department notes, radiology reports, and discharge summaries. The lexicon and classification logic were tailored for each note type. The system was first developed and evaluated using annotated notes from the Department of Veterans Affairs. Interoperability was assessed using data from the University of Utah. ResultsThe NLP system was comprised of 782 rules and achieved moderate-to-high performance in all three note types in VA (precision/recall/f1: emergency=88.1/86.0/87.1; radiology=71.4/96.2/82.0; discharge=88.3/93.0/90.1). When applied to UU data, performance was maintained in emergency and radiology but decreased in discharge summaries (emergency=84.7/94.3/89.3; radiology=79.7/100.0/87.9; discharge=65.5/92.7/76.8). Customization with 34 additional rules increased performance for all note types (emergency=89.3/94.3/91.7; radiology=87.0/100.0/93.1; discharge=75.0/95.1/83.4). ConclusionNLP can be used to accurately identify the diagnosis of pneumonia in different clinical settings and institutions. A limited amount of customization to account for differences in lexicon, clinical definition of pneumonia, and EHR structure can achieve high accuracy without substantial modification.