Photoacoustics-guided Real-Time Closed-loop Control of Magnetic Microrobots through Deep Learning

Medical microrobots promise to increase the efficacy and reduce the invasiveness of certain medical procedures in the future. Real-time tracking of the microrobot, actuation, and closed-loop control of its position under in vivo conditions is crucial to fulfill the task at hand. We present a system for closed-loop control of magnetic microrobots using dual-mode ultrasound and photoacoustic imaging. It employs GPU-accelerated beamforming and tracking to achieve real-time operation with a closed-loop cycle time of 100 ms. Artifacts from simultaneous imaging and magnetic actuation are suppressed through time-multiplexing. To address the challenge of detecting microrobots in low-contrast, strong-background images, we implemented real-time Deep Learning-based tracking. A custom dataset of various types of microrobots is curated from long-duration closed-loop control measurements and employed to fine-tune a pre-trained detection model. We introduce a platform for real-time closed-loop control of microrobots and demonstrate its performance with a 300 m spiral-shaped microrobot following a figure-of-8 shape under photoacoustic imaging guidance. The localization error is evaluated against an optical reference measurement. Our results show that photoacoustic-based tracking significantly outperforms ultrasound tracking, with the deep learning approach further reducing missed detections. This demonstrates the algorithms ability to generalize to a previously unseen type of microrobot. We envision this platform to advance medical microrobotics research by providing real-time closed-loop control of untethered microrobots under deep tissue.