From Blurry to Brilliant: HAGAN, a Hybrid Attention GAN for Home-Based OCT Image Enhancement with Magical Results

Regular optical coherence tomography (OCT) monitoring is essential for early detection of retinal disease and timely intervention, but frequent clinicbased imaging burdens patients and healthcare systems. Home-based OCT enables continuous monitoring and reduces clinic visits; however, compact optics and patient-operated acquisition introduce noise, reduced resolution, motion blur, and artifacts that limit clinical reliability and diagnostic confidence. To model home-based OCT acquisition, we employ simulated data reflecting images from Siloton, a compact home-based OCT device. Clinically realistic noise and acquisition artifacts were applied to high-quality OCT images using Silotons simulation software, generating near-real patient-operated scans. Building on this dataset, we propose HAGAN, a Hybrid Attention Generative Adversarial Network developed through a progressive strategy, evolving from a baseline U-Net to an adversarial framework with hybrid attention. The best-performing U-Net architecture, EfficientNet-B1, identified through evaluation and ablation studies, is adopted as the generator. The generator incorporates attention gates at its skip connections and self-attention modules within the decoder, and is paired with a VGG19-based discriminator to form the HAGAN architecture. The model is trained using a multiobjective loss combining pixel-wise, structural, perceptual, edge-preserving, and adversarial components. Experiments on simulated home-based OCT data demonstrate that HAGAN consistently outperforms baseline and state-of-the-art models across standard enhancement metrics and a clinically relevant retinal layer segmentation downstream task, improving visual quality and preservation of diagnostically meaningful anatomical structures. These findings support the potential of HAGAN for reliable enhancement in future home-based OCT platforms, enabling remote retinal monitoring and reducing reliance on in-clinic imaging and routine hospital visits. HighlightsO_LIEnhancing the quality of home-based OCT images to support remote retinal monitoring and reduce the need for frequent referrals to clinical imaging centers C_LIO_LIProposing HAGAN, a hybrid attention generative adversarial network for enhancing OCT images acquired using the Siloton home-based OCT device C_LIO_LIHybrid attention design combining attention gates and self-attention to preserve fine retinal details and global anatomical consistency C_LIO_LIAdversarial learning framework improving perceptual realism and preservation of diagnostically relevant retinal structures in low-quality homeacquired OCT images C_LIO_LIProgressive model development from baseline U-Net to hybrid attention GAN, demonstrating systematic and measurable performance improvements C_LIO_LIClinical relevance validated through downstream retinal layer segmentation, confirming preservation of diagnostically important structures C_LI