Back

Retrospective Evaluation of a Generative AI-Enabled Electronic Medical Record System in Primary Health Care Facilities in Kenya

Agweyu, A.; Mwaniki, P.; Musau, W.; Korom, R.; Isaaka, L.; Wanyama, C.; kiptinness, S.; Adan, N.; Emmanual-Fabula, M.; Mateen, B.

2025-09-07 primary care research
10.1101/2025.09.05.25335163 medRxiv
Show abstract

We conducted a retrospective evaluation of an electronic medical record-embedded large language model (LLM) clinical decision support system deployed across 16 primary care clinics in Kenya, between July-September 2024. A panel of trained physicians reviewed 1,469 records. Hallucinations were uncommon (50/1,469; 3.4%), most often involving mis-expanded acronyms or drug names. Clinical management guidance aligned with local guidelines in almost all cases (approximately 100%). Despite this, clinicians did not modify documentation in 62% of encounters. Safety assessments identified actively harmful recommendations from the LLM in 7.8% of encounters, with 67 such recommendations appearing in the final documentation. Conversely, risk present in the clinicians initial notes was fully mitigated in 118 encounters (8.0% overall; 12.1% of amended cases). Overall, the tool showed strong potential to support quality improvement, but the asymmetric adoption of harmful versus beneficial outputs underscore the need for usability optimization, local guardrails, and prospective trials to confirm patient-level benefit.

Matching journals

The top 6 journals account for 50% of the predicted probability mass.

1
PLOS Global Public Health
● 344 papers in training set
Top 0.9%
18.6%
Similar papers in this journal
2
Journal of the American Medical Informatics Association
◐ 71 papers in training set
Top 0.3%
9.8%
Similar papers in this journal
3
npj Digital Medicine
○ 118 papers in training set
Top 0.8%
7.3%
Similar papers in this journal
4
The Lancet Digital Health
○ 25 papers in training set
Top 0.1%
6.8%
Similar papers in this journal
5
BMJ Health & Care Informatics
● 15 papers in training set
Top 0.1%
6.8%
Similar papers in this journal
6
PLOS ONE
● 5266 papers in training set
Top 27%
5.5%
50% of probability mass above
7
British Journal of General Practice
● 23 papers in training set
Top 0.1%
4.1%
8
BMJ Open
● 601 papers in training set
Top 6%
4.1%
9
Communications Medicine
○ 113 papers in training set
Top 0.6%
3.4%
10
Journal of Medical Internet Research
◐ 87 papers in training set
Top 0.8%
3.3%
11
BJGP Open
● 13 papers in training set
Top 0.1%
3.2%
12
PLOS Digital Health
● 106 papers in training set
Top 2%
2.4%
13
BMC Medicine
○ 176 papers in training set
Top 2%
2.1%
14
eLife
● 5828 papers in training set
Top 45%
2.1%
15
Health Policy
○ 11 papers in training set
Top 0.5%
1.0%
16
Epidemiology
○ 32 papers in training set
Top 0.5%
0.9%
17
BMC Health Services Research
○ 51 papers in training set
Top 2%
0.8%
18
The American Journal of Tropical Medicine and Hygiene
● 68 papers in training set
Top 2%
0.8%
19
JMIR Public Health and Surveillance
◐ 45 papers in training set
Top 2%
0.8%
20
JMIR Medical Informatics
◐ 18 papers in training set
Top 0.8%
0.8%
21
JAMA Network Open
● 130 papers in training set
Top 4%
0.8%
22
Pilot and Feasibility Studies
○ 14 papers in training set
Top 0.7%
0.8%
23
Frontiers in Digital Health
○ 24 papers in training set
Top 1%
0.8%
24
JMIR Research Protocols
◐ 21 papers in training set
Top 1%
0.8%
25
Healthcare
○ 17 papers in training set
Top 1%
0.6%
26
JAMA
● 18 papers in training set
Top 0.5%
0.6%
27
Cancer Medicine
○ 26 papers in training set
Top 1%
0.6%
28
The Journal of Pediatrics
○ 16 papers in training set
Top 0.4%
0.6%

"Similar papers" are the closest papers from that journal in the model's embedding space. They show what the match is built on, but the ranking comes mostly from a classifier over the whole training set, not from these examples alone.