Play it by Ear: A perceptual algorithm for autonomous melodious piano playing with a bio-inspired robotic hand

Perception shapes the learning and performance of motor behavior in animals. In contrast to this inherent biological and psychological connection between perception and action, traditional artificial intelligence methods for robotics emphasize reward-driven extensive trial-and-error or error-driven control techniques. Our study goes back to the perceptual roots of biological learning and behavior, and demonstrates a novel end-to-end perceptual experience-driven approach for autonomous piano playing. Our Play it by Ear perceptual learning algorithm, coupled to a bio-inspired 4-finger robotic hand, can replicate melodies on a keyboard after hearing them once--without explicit or prior knowledge of notes, the hand, or the keyboard. Our key innovation is an end-to-end pipeline that, after a brief period of motor babbling by the hand, converts the sound of a melody into native musical percepts (note sequences and intensities) that it replays as sequences of key presses. In this way, any new melody consisting of notes experienced during babbling can be reproduced by the robotic musician hand on the basis of its percepts. This playback includes capturing the qualitative and quantitative musical dynamics and tempo with a nuance comparable with that of four human pianists performing the same melody. These compelling results emphasize the perceptual underpinnings of artistic performance as an alternative to traditional control-theoretical emphasis on state estimation and error correction. Our approach opens avenues for the development of simple machines that can still execute artistic and physical tasks that approach the nuance inherent in human behavior.