Integrated data processing between image and audio - Musical instrument (piano) playing information processing

This paper proposes an integrated data processing system between video and audio. The subject in this paper is the action of "Playing the Piano". First the information generated from the action is defined according to the meanings; scale, key etc. Then depending on the meanings, each information is processed in a media space, where it is likely easier processed. We designed a co-operative musical scale detection system of audio and video data and performed basic experiments. The simulation results show that image processing only gave the detection rate over 90 %(white-key) and sound processing could cope with faster key-hit movement and black-key hit detection and combining 2 processes gave higher detection rate. The proposed method can be applied to audio data compression. With the simple sound description model (Musical Scale, Amplitude and Phase), about 40-70 % of data reduction was achieved with better quality than processing audio alone.