PANDORA: An AI model for the automatic extraction of clinical unstructured data and clinical risk score implementation

IntroductionMedical records and physician notes often contain valuable information not organized in tabular form and usually require extensive manual processes to extract and structure. Large Language Models (LLMs) have shown remarkable abilities to understand, reason, and retrieve information from unstructured data sources (such as plain text), presenting the opportunity to transform clinical data into accessible information for clinical or research purposes. ObjectiveWe present PANDORA, an AI system comprising two LLMs that can extract data and use it with risk calculators and prediction models for clinical recommendations as the final output. MethodsThis study evaluates the models ability to extract clinical features from actual clinical discharge notes from the MIMIC database and synthetically generated outpatient clinical charts. We use the PUMA calculator for Chronic Obstructive Pulmonary Disease (COPD) case finding, which interacts with the model and the retrieved information to produce a score and classify patients who would benefit from further spirometry testing based on the 7 items from the PUMA scale. ResultsThe extraction capabilities of our model are excellent, with an accuracy of 100% when using the MIMIC database and 99% for synthetic cases. The ability to interact with the PUMA scale and assign the appropriate score was optimal, with an accuracy of 94% for both databases. The final output is the recommendation regarding the risk of a patient suffering from COPD, classified as positive according to the threshold validated for the PUMA scale of equal to or higher than 5 points. Sensitivity was 86% for MIMIC and 100% for synthetic cases. ConclusionLLMs have been successfully used to extract information in some cases, and there are descriptions of how they can recommend an outcome based on the researchers instructions. However, to the best of our knowledge, this is the first model which successfully extracts information based on clinical scores or questionnaires made and validated by expert humans from plain, non-tabular data and provides a recommendation mixing all these capabilities, using not only knowledge that already exists but making it available to be explored in light of the highest quality evidence in several medical fields.