Human-Robot Percussion Ensemble Anticipation on the Basis of Visual Cues

The importance of visual gesture anticipation in music performance involving humans and robots is studied through computer vision. The anticipation is studied using the context of a percussionist playing a set of four drums using mallets. The Nintendo Wii Remote controller is used to track the bidimensional coordinates of the mallets on the scene. Small opaque spheres are placed around the LEDs so that the emitted light spreads in a wider solid angle. As the main gesture performed by the drummer is the movement of his or her arm, from the moment he or she starts lifting the mallet to the moment that sound is produced, researchers are interested in detecting when such a gesture starts so that we can predict which mallet should be hit and at what time in the future the sound should be produced. Because the mallet must descend toward the drum at a certain minimum speed to make the right sound, the time of anticipation achieved with the first approach is not enough to circumvent the delay between the instruction to play and the action of playing by the robot.