Accents Still Confuse AI: Systematic Errors in Speech Transcription and LLM-Based Remedies
Fatapour, Y.; Samaan, J. S.; Kuchi, A.; Srinivasan, A. P.; Fatapour, S.; Liu, H.; Berkowitz, J. S.; Tsang, K.; Zietz, M.; Friedrich, N.; Srinivasan, N.; Thangaratnam, S.; King, R.; Czarny, R.; Nguyen, T.; Yeo, Y. H. S.; Kim, H.; Lee, Y.-T.; Wongjarupong, N.; Abiri, A.; Tatonetti, N. P.
Show abstract
Accurate and timely documentation in the electronic health record (EHR) is essential for delivering safe and effective patient care. AI-enabled medical tools powered by automatic speech recognition (ASR) offer to streamline this process by transcribing clinical conversations directly into structured notes. However, a critical challenge in deploying these technologies at scale is their variable performance across speakers with diverse accents, which leads to transcription inaccuracies, misinterpretation, and downstream clinical risks. We measured transcription accuracy of Whisper and WhisperX on clinical texts across native and non-native English speakers and found that both models have significantly higher errors for non-native speakers. Fortunately, we found that post-processing the transcripts using GPT-4o recovers the lost accuracy. Our findings indicate that using a chained model approach, WhisperX-GPT, will enhance transcription quality significantly and reduce errors associated with accented speech. We make all code, models, and pipelines freely available.
Matching journals
The top 6 journals account for 50% of the predicted probability mass.