Comparison of ChatGPT and Gemini as sources of references in otorhinolaryngology
Jedrzejczak, W. W.; Pastucha, M.; Skarzynski, H.; Kochanek, K.
Show abstract
IntroductionAn effective way of testing chatbots is to ask them for references since such items can be easily verified. The purpose of this study was to compare the ability of ChatGPT-4 and Gemini Advanced to select accurate references on common topics in otorhinolaryngology. MethodsChatGPT-4 and Gemini Advanced were asked to provide references on 25 topics within the otorhinolaryngology category of Web of Science. Within each topic, we set as target the most cited papers which had "guidelines" in the title. The chatbot responses were collected on three consecutive days to take into account possible variability. The accuracy and reliability of the provided references were evaluated. ResultsAcross the three days, the accuracy of ChatGPT-4 was 29-45% while that of Gemini Advanced was 10-17%. Common errors included false author names, false DOI numbers, and incomplete information. Lower percentage errors were associated with higher number of citations. ConclusionsBoth chatbots performed poorly in finding references, although ChatGPT-4 provided higher accuracy than Gemini Advanced.
Matching journals
The top 3 journals account for 50% of the predicted probability mass.
Similar papers in this journal
- Attitudes and Practices of Open Data, Preprinting, and Peer-review - a Cross Sectional Study on Croatian Scientists 95%
- Investigating the Nature of Open Science Practices Across Complementary, Alternative, and Integrative Medicine Journals: An Audit 95%
- Woman Authorship in Pre-print Versus Peer-Reviewed Oral Health-Related Publications: A Two-Year Observational Study 94%
Similar papers in this journal
- Performance of ChatGPT in pediatric audiology as rated by students and experts 96%
- Artificial Intelligence in laryngeal endoscopy: Systematic Review and Meta-Analysis 94%
- A Call For Better Methodological Quality Of Reviews On Using Artificial Intelligence For COVID-19 Detection In Medical Imaging – An Umbrella Systematic Review 92%
Similar papers in this journal
- Google Trends as a predictive tool for COVID-19 vaccinations in Italy: a retrospective infodemiological analysis 91%
- Learning from the resilience of hospitals and their staff to the COVID-19 pandemic: a scoping review 91%
- Caregivers’ Perspective: Satisfaction With Healthcare Services At The Paediatric Specialist Clinic Of The National Referral Centre In Malaysia 91%
Similar papers in this journal
- Researcher and Clinician Preferences for a Journal Transparency Tool: A Mixed-Methods Survey and Focus Group Study 95%
- Evaluation of the performance of GPT-3.5 and GPT-4 on the Medical Final Examination 94%
- Content-based image retrieval assists radiologists in diagnosing eye and orbital mass lesions in MRI 92%
Similar papers in this journal
- Partition Quantitative Assessment (PQA): A quantitative methodology to assess the embedded noise in clustered omics and systems biology data 91%
- An image processing protocol to extract variables predictive of human embryo fitness for assisted reproduction 89%
- SelectStitch: Automated Frame Segmentation and Stitching to Create Composite Images from Otoscope Video Clips 89%
"Similar papers" are the closest papers from that journal in the model's embedding space. They show what the match is built on, but the ranking comes mostly from a classifier over the whole training set, not from these examples alone.