Performance of Frangi-Hessian Pseudo-Labels for Retinal Vessel Segmentation in AI-Assisted Retinopathy of Prematurity Screening

BackgroundRetinopathy of prematurity (ROP) is a leading cause of preventable blindness among preterm infants. Accurate retinal vessel segmentation is crucial for detecting plus disease, which indicates progression to severe ROP. However, manual annotation of vessel masks is laborious and inconsistent, especially in low-resource clinical settings. This study aimed to evaluate a self-supervised vessel extraction pipeline using Frangi-Hessian filtering for automatic pseudo-annotation of unlabeled RetCam and Neo retinal images and to compare its performance against supervised and hybrid deep learning frameworks. MethodsTwo public datasets from the HVDROPDB-BV repository: RetCam_Vessels and Neo_Vessels were utilized. We implemented a three-stage pipeline: automatic self-annotation of unlabeled images through vessel-based mask generation; training of five segmentation architectures--BioSwinFuseNet, UNet, FPN, LinkNet, and SegFormer--under three regimes (GT-only, Self-only, and Hybrid GT+Self); and evaluation using Dice, IoU, sensitivity, specificity, PPV, NPV, F1, and AUC metrics. All models were trained with a topology-aware loss that combined binary cross-entropy and Dice losses with continuity penalties. ResultsHybrid supervision consistently outperformed both GT-only and Self-only training across all architectures. The SegFormer-Hybrid model achieved the highest Dice (0.61) and IoU (0.44), while FPN-Hybrid demonstrated the lowest variance. BioSwinFuseNet-Hybrid showed a 122% relative improvement in Dice compared to its GT-only version. Self-only models learned rudimentary vessel priors but lacked clinical precision. ConclusionsIncorporating self-annotated masks alongside limited ground truth improves segmentation accuracy and vessel continuity. The hybrid paradigm offers a scalable path for developing automated ROP screening tools where expert labeling is limited.