BioLP-bench: Measuring understanding of biological lab protocols by large language models
Ivanov, I.
Show abstract
Language models rapidly become more capable in many domains, including biology. Both AI developers and policy makers [1] [2] [3] are in need of benchmarks that evaluate their proficiency in conducting biological research. However, there are only a handful of such benchmarks[4, 5], and all of them have their limitations. This paper introduces the Biological Lab Protocol benchmark (BioLP-bench) that evaluates the ability of language models to find and correct mistakes in a diverse set of laboratory protocols commonly used in biological research. To evaluate understanding of the protocols by AI models, we introduced in these protocols numerous mistakes that would still allow them to function correctly. After that we introduced in each protocol a single mistake that would cause it to fail. We then presented these modified protocols to an LLM, prompting it to identify the mistake that would cause it to fail, and measured the accuracy of a model in identifying such mistakes across many test cases. Only OpenAI o1-preview scored similarly to the performance of human experts, while other language models demonstrated substantially worse performance, and in most cases couldnt correctly identify the mistake. Code and dataset are published at https://github.com/baceolus/BioLP-bench
Matching journals
The top 8 journals account for 50% of the predicted probability mass.
Similar papers in this journal
- Countering reproducibility issues in mathematical models with software engineering techniques: A case study using a one-dimensional mathematical model of the atrioventricular node 94%
- The Mastery Rubric for Bioinformatics: supporting design and evaluation of career-spanning education and training 94%
- Leveraging large language models for data analysis automation 94%
Similar papers in this journal
Similar papers in this journal
- Open-loop lab-on-a-chip technology enables remote computer science training in Latinx life sciences students 93%
- AutoGater: A Weakly Supervised Neural Network Model to Gate Cells in Flow Cytometric Analyses 92%
- An Assistive Computer Vision Tool to Automatically Detect Changes in Fish Behavior In Response to Ambient Odor 92%
Similar papers in this journal
"Similar papers" are the closest papers from that journal in the model's embedding space. They show what the match is built on, but the ranking comes mostly from a classifier over the whole training set, not from these examples alone.