Companion Website for:

Perception in action: a robotic system that can teach itself to melodiously play music by ear

Journal of The Royal Society Interface - May 2026

Link to PDF version from Valero Lab
Link to J R Soc Interface Article
Link to Viterbi Press Release

Learning to imitate nuanced motor behaviour by perceiving someone’s actions is a human ability central to non‑verbal communication, interaction and collaboration. Despite extensive psychophysics research, robotic systems do not yet seamlessly bridge the gap between perception and action. We demonstrate a perceptual learning algorithm that can replicate a melody after hearing it once by using a physical robotic hand on a keyboard. Importantly, this system (the Musician Hand) can do this after only 2 min of random ‘motor babbling’, and driven only by its own experience, with minimal pre‑defined features of musical notes. Experiments with three melodies showed that our system can play‑by‑ear at a level comparable with four trained pianists, and better than five novices. This demonstrates how perception can seamlessly drive nuanced robotic action based only on its own limited experience. This demonstration of a perception‑driven system paves the way for human‑friendly and intuitive systems for entertainment, collaboration and physical assistance/augmentation.

Citation

Azadjou, H., Marjaninejad, A., & Valero-Cuevas, F. J. (2026); Perception in action: a robotic system that can teach itself to melodiously play music by ear. J R Soc Interface, 23 (238): 20250909.

Acknowledgements

We had the privilege of collaborating with two composers, Richard Tuttobene and Targol Karimi Moghaddam, who graciously crafted three new melodies. These pieces capture melodious musical dynamics using only four adjacent notes. We also thank Jan Lao and Zeyu (Claude) Yao, who helped us build the hardware for the Musician Hand. Special thanks to Angelo Bartch‑Jimenez and Majid Abbasi‑Sisara who contributed to this work with their comments and insights.

For questions and more information, please send us an email: [email protected]