Structured extraction from Russian/Kazakh discharge summaries: three model families at the human-agreement ceiling
Aimyshev, T.; Arupzhanov, I.; Zhakhina, G.; Rakhmankulova, A.; Smagul, S.; Gaipov, A.
Show abstract
Objective. To determine which model family a hospital should use to extract structured data (sections, medication records, laboratory tuples) from Russian/Kazakh stroke discharge summaries, bounded against measured human agreement. Materials and Methods. Three families were compared under identical inputs: a multilingual BERT-style encoder with CRF heads, a locally fine-tuned Qwen3-4B, and zero-shot GPT-5.5, on three tasks of increasing granularity: section segmentation (4,327 annotated documents), seven-field medication records (708 documents), and dated lab tuples (1,888 documents). All arms shared a block-ID representation (extraction as tagging over a fixed segmentation). Blind re-annotation of 50 documents per task provided the human-agreement ceiling; model-vs-ceiling claims use paired same-document tests with Holm correction. Results. On section segmentation all arms fall within {+/-}0.03 of the ceiling (0.905; models 0.919-0.932). On medication detection the cloud model is above it (0.966 vs 0.959); on full-record assembly, re-scored symmetrically against a blind annotator as common reference, both the fine-tuned LLM and the cloud model reach 0.344 agreement versus the gold annotator's 0.183. On lab assembly the fine-tuned LLM matches the full-tuple ceiling (0.898 vs 0.902) while the encoder is far below (0.476). Residual errors concentrate in annotation conventions the annotators themselves could not agree on. Discussion. Detection is solved at the human ceiling even for non-English clinical text, and a minutes-scale fine-tune reaches it with no data egress. Assembly separates the families and is convention-bound: benchmark consumers should read "above human agreement" against what the guidelines fixed. Conclusion. Annotation guidelines and task representation, not model capacity, are the binding constraints for this task family.
Matching journals
The top 6 journals account for 50% of the predicted probability mass.
Similar papers in this journal
- Measuring the Quality of AI-Generated Clinical Notes: A Systematic Review and Experimental Benchmark of Evaluation Methods 92%
- Comparing neural language models for medical concept representation and patient trajectory prediction 92%
- Building Large-Scale Registries from Unstructured Clinical Notes using a Low-Resource Natural Language Processing Pipeline 91%
Similar papers in this journal
Similar papers in this journal
- Annotation-preserving machine translation of English corpora to validate Dutch clinical concept extraction tools 94%
- LCD Benchmark: Long Clinical Document Benchmark on Mortality Prediction for Language Models 93%
- Generative Large Language Models in Electronic Health Records for Patient Care Since 2023: A Systematic Review 92%
Similar papers in this journal
- Large Language Models for Zero-Shot Procedure Extraction in Orthopedic Surgery: A Comparative Evaluation 93%
- CONSORT-TM: Text classification models for assessing the completeness of randomized controlled trial publications 91%
- EHR Foundation Models Improve Robustness in the Presence of Temporal Distribution Shift 91%
Similar papers in this journal
- MelAnalyze: Fact-Checking Melatonin claims using Large Language Models and Natural Language Inference 92%
- Large language models (GPT-5, Grok-4, Claude Opus 4.1, Gemini 2.5 Pro) achieved textbook-level accuracy on the Japanese medical licensing examination by 2025: A comparative study 91%
- Evaluating Semantic Similarity Methods for Comparison of Text-derived Phenotype Profiles 90%
"Similar papers" are the closest papers from that journal in the model's embedding space. They show what the match is built on, but the ranking comes mostly from a classifier over the whole training set, not from these examples alone.