Back

Can AI Match Human Experts? Evaluating LLM-Generated Feedback on Resident Scholarly Projects

2026-03-04 medical education Title + abstract only
View on medRxiv
Show abstract

BackgroundDelivering timely, high-quality feedback on resident scholarly projects is labour-intensive, especially in large programmes. We developed an AI-assisted evaluation system, powered by the open-weight LLaMA-3.1 large-language model (LLM), to generate formative feedback on Family Medicine residents scholarly projects and compared its performance with expert human evaluators. MethodsWe evaluated whether the AI-generated feedback achieves comparable quality to expert feedback. The tool ing...

Predicted journal destinations

1
BMC Medical Education
16 training papers
Top 0.5% 16.0%
2
PLOS ONE
1737 training papers
Top 44% 16.0%
3
PLOS Digital Health
88 training papers
Top 0.4% 13.0%
4
Journal of Medical Internet Research
81 training papers
Top 0.7% 9.9%
5
BMJ Open
553 training papers
Top 31% 5.5%
6
npj Digital Medicine
85 training papers
Top 6% 4.2%
7
Scientific Reports
701 training papers
Top 77% 2.1%
8
Cureus
64 training papers
Top 8% 1.7%
9
Frontiers in Public Health
135 training papers
Top 18% 1.7%
10
Computers in Biology and Medicine
39 training papers
Top 5% 1.5%
11
The Lancet Digital Health
25 training papers
Top 0.4% 1.5%
12
Journal of Clinical Epidemiology
29 training papers
Top 1% 1.5%
13
Biology Methods and Protocols
19 training papers
Top 0.5% 1.3%
14
JAMA
17 training papers
Top 0.2% 1.0%
15
Annals of Internal Medicine
27 training papers
Top 0.5% 1.0%
16
International Journal of Medical Informatics
25 training papers
Top 6% 0.8%
17
PLOS Global Public Health
287 training papers
Top 32% 0.8%
18
Journal of the American Medical Informatics Association
53 training papers
Top 6% 0.8%
19
JAMA Network Open
125 training papers
Top 28% 0.8%
20
Research Synthesis Methods
17 training papers
Top 0.7% 0.8%
21
Journal of General Internal Medicine
19 training papers
Top 2% 0.6%
22
Heliyon
57 training papers
Top 20% 0.5%
23
PLOS Computational Biology
141 training papers
Top 17% 0.5%
24
Nature Communications
483 training papers
Top 65% 0.5%
25
Journal of Biomedical Informatics
37 training papers
Top 6% 0.5%
26
eLife
262 training papers
Top 65% 0.5%
27
BMC Medicine
155 training papers
Top 42% 0.5%