Human-Robot Cooperative Piano Playing With Learning-Based Real-Time Music Accompaniment

被引:0
作者
Wang, Huijiang [1 ]
Zhang, Xiaoping [1 ]
Iida, Fumiya [1 ,2 ]
机构
[1] Univ Cambridge, Dept Engn, Bioinspired Robot Lab, Cambridge CB2 1PZ, England
[2] North China Univ Technol, Sch Elect & Control Engn, Beijing 100144, Peoples R China
关键词
Robots; Robot kinematics; Robot sensing systems; Real-time systems; Synchronization; Manipulators; Harmonic analysis; Collaborative robots; Skin; End effectors; Collaborative manipulation; entertainment robot; human-robot cooperation; piano playing; transfer entropy; MODELS;
D O I
10.1109/TRO.2024.3486408
中图分类号
TP24 [机器人技术];
学科分类号
080202 ; 1405 ;
摘要
Recent advances in machine learning have paved the way for the development of musical and entertainment robots. However, human-robot cooperative instrument playing remains a challenge, particularly due to the intricate motor coordination and temporal synchronization. In this article, we propose a theoretical framework for human-robot cooperative piano playing based on nonverbal cues. First, we present a music improvisation model that employs a recurrent neural network (RNN) to predict appropriate chord progressions based on the human's melodic input. Second, we propose a behavior-adaptive controller to facilitate seamless temporal synchronization, allowing the cobot to generate harmonious acoustics. The collaboration takes into account the bidirectional information flow between the human and robot. We have developed an entropy-based system to assess the quality of cooperation by analyzing the impact of different communication modalities during human-robot collaboration. Experiments demonstrate that our RNN-based improvisation can achieve a 93% accuracy rate. Meanwhile, with the MPC adaptive controller, the robot could respond to the human teammate in homophony performances with real-time accompaniment. Our designed framework has been validated to be effective in allowing humans and robots to work collaboratively in the artistic piano-playing task.
引用
收藏
页码:4650 / 4669
页数:20
相关论文
共 51 条
[1]   Progress and prospects of the human-robot collaboration [J].
Ajoudani, Arash ;
Zanchettin, Andrea Maria ;
Ivaldi, Serena ;
Albu-Schaeffer, Alin ;
Kosuge, Kazuhiro ;
Khatib, Oussama .
AUTONOMOUS ROBOTS, 2018, 42 (05) :957-975
[2]  
Akella P, 1999, ICRA '99: IEEE INTERNATIONAL CONFERENCE ON ROBOTICS AND AUTOMATION, VOLS 1-4, PROCEEDINGS, P728, DOI 10.1109/ROBOT.1999.770061
[3]   Controlling Ocean One: Human-robot collaboration for deep-sea manipulation [J].
Brantner, Gerald ;
Khatib, Oussama .
JOURNAL OF FIELD ROBOTICS, 2021, 38 (01) :28-51
[4]   Collaborative manufacturing with physical human-robot interaction [J].
Cherubini, Andrea ;
Passama, Robin ;
Crosnier, Andre ;
Lasnier, Antoine ;
Fraisse, Philippe .
ROBOTICS AND COMPUTER-INTEGRATED MANUFACTURING, 2016, 40 :1-13
[5]   The relevance of signal timing in human-robot collaborative manipulation [J].
Cini, F. ;
Banfi, T. ;
Ciuti, G. ;
Craighero, L. ;
Controzzi, M. .
SCIENCE ROBOTICS, 2021, 6 (58)
[6]   A survey of multi-agent Human-Robot Interaction systems [J].
Dahiya, Abhinav ;
Aroyo, Alexander M. ;
Dautenhahn, Kerstin ;
Smith, Stephen L. .
ROBOTICS AND AUTONOMOUS SYSTEMS, 2023, 161
[7]  
Dan Zhang, 2009, 2009 International Conference on Information and Automation (ICIA), P757, DOI 10.1109/ICINFA.2009.5205022
[8]   A Hierarchical Architecture for Human-Robot Cooperation Processes [J].
Darvish, Kourosh ;
Simetti, Enrico ;
Mastrogiovanni, Fulvio ;
Casalino, Giuseppe .
IEEE TRANSACTIONS ON ROBOTICS, 2021, 37 (02) :567-586
[9]   Jamming Joints for Stiffness and Posture Control with an Anthropomorphic Hand [J].
Gilday, Kieran ;
Hughes, Josie ;
Iida, Fumiya .
2021 IEEE 4TH INTERNATIONAL CONFERENCE ON SOFT ROBOTICS (ROBOSOFT), 2021, :134-140
[10]   Human-robot interaction: A survey [J].
Brigham Young University, Provo, UT 84602, United States ;
不详 .
Found. Trends Human-Comput. Interact., 2007, 3 (203-275)