Multimodal AI agents for capturing and sharing laboratory practice
Skowronek, P.; Nawalgaria, A.; Mann, M.
Show abstract
We present a multimodal AI laboratory agent that captures and shares tacit experimental practice by linking written instructions with hands-on laboratory work through the analysis of video, speech, and text. While current AI tools have proven effective in literature analysis and code generation, they do not address the critical gap between documented knowledge and implicit lab practice. Our framework bridges this divide by integrating protocol generation directly from researcher-recorded videos, systematic detection of experimental errors, and evaluation of instrument readiness by comparing current performance against historical decisions. Evaluated in mass spectrometry-based proteomics, we demonstrate that the agent can capture and share practical expertise beyond conventional documentation and identify common mistakes, although domain-specific and spatial recognition should still be improved. This agentic approach enhances reproducibility and accessibility in proteomics and provides a generalizable model for other fields where complex, hands-on procedures dominate. This study lays the groundwork for community-driven, multimodal AI systems that augment rather than replace the rigor of scientific practice.
Matching journals
The top 5 journals account for 50% of the predicted probability mass.
Similar papers in this journal
Similar papers in this journal
- An adaptive, continuous-learning framework for clinical decision-making from proteome-wide biofluid data 94%
- Imputation of label-free quantitative mass spectrometry-based proteomics data using self-supervised deep learning 94%
- Trackable and scalable LC-MS metabolomics data processing using asari 93%
Similar papers in this journal
- A Simple Cross-Linking/Mass Spectrometry Workflow to Study System-Wide Protein Interactions 93%
- A universal spectrum annotator for complex peptidoforms in mass spectrometry-based proteomics 92%
- Rapid QC-MS - Interactive Dashboard for Synchronous Mass Spectrometry Data Acquisition Quality Control 92%
"Similar papers" are the closest papers from that journal in the model's embedding space. They show what the match is built on, but the ranking comes mostly from a classifier over the whole training set, not from these examples alone.