Assessing Multimodal AI for Visual Information Extraction of Pharmacology
Dilan-Pantojas, I. O.; Duong, P. T.; Lopes, K.; Boyce, R. D.
Show abstract
While Americans are using herbal dietary supplements (natural products) more than ever, the consumption of natural products with prescription drugs can lead to harmful interactions. Pharmacovigilance of natural products depends on careful expert review and interpretation of a wide variety of evidence. In prior work, we demonstrated the value of knowledge graph (NP-KG) for assisting with natural product safety investigations. However, scaling the NP-KG from 33 natural products to the thousands on the market requires computer-assisted data extraction, particularly from visual elements (figures or tables) of pharmacology literature. We evaluated the accuracy and resilience of 8 open- and closed-source multimodal models by performing visual information extraction from select tables and images. The best performing models could accurately extract 90% of tabular data and 45% of data reported figures with a modified relative error rate of 0.05. Image resolution and information density were primary hindrances to better extraction performance.
Matching journals
The top 8 journals account for 50% of the predicted probability mass.
Similar papers in this journal
Similar papers in this journal
- Evaluating Knowledge Fusion Models on Detecting Adverse Drug Events in Text 93%
- Ethical review of clinical research with generative AI: Evaluating ChatGPT’s accuracy and reproducibility 93%
- Inferring Gender from First Names: Comparing the Accuracy of Genderize, Gender API, and the gender R Package on Authors of Diverse Nationality 93%
Similar papers in this journal
Similar papers in this journal
"Similar papers" are the closest papers from that journal in the model's embedding space. They show what the match is built on, but the ranking comes mostly from a classifier over the whole training set, not from these examples alone.