Refined shoulder kinematics via markerless bony landmark detection and acromial 3D shape using an RGB-D camera during hand-cycling

Biomechanical biofeedback has the potential to enhance rehabilitation by providing clinicians with objective evaluation of patient performances. As feedback systems often depend on expensive and sophisticated motion capture technologies, researchers explore computer vision-based alternatives. Existing methods suffer from substantial joint angle errors, particularly in the upper limb, and neglect the scapular movements. We developed an approach for detecting bony landmarks and performing refined upper-limb kinematics assessments using a single consumer-grade depth-sensing camera. Unlike other markerless methods, our model incorporates the scapula, offering comprehensive shoulder joint kinematics. Annotated images from eight participants were used to fine-tune a convolutional neural network, which was subsequently evaluated on a hand-cycling motion. Our method showed a strong agreement with a reference marker-based system, with 3D bony landmark detection errors averaging 5 mm. The resulting kinematics closely aligned with the reference system, maintaining acceptable joint angle errors ([~]6.3{degrees}). Furthermore, the algorithm could provide real-time bony landmark positions and joint kinematics at a rate of 50 Hz. This study highlights the potential of using a single consumer-grade depth-sensing camera combined with an acromial 3D-shape to accurately estimate upper-limb kinematics through bony landmark detection, paving the way for more accessible clinical assessments.