System for multimodal data acquisition for human action recognition

Multimodal data is being used more widely for human action recognition nowadays due to the progress of machine learning methods and the development of new types of sensors. The acquisition of the data required by such solutions is often troublesome, and it is difficult to find the proper tools for this process. In this paper, we present a new toolkit for multimodal acquisition. We address and discuss issues concerning the synchronization of data from multiple sensors, the optimization of the initial processing of raw data, and the design of the user interface for efficiently recording large databases. The system was verified in a setup consisting of three types of sensors – a Kinect 2, two PS3Eye cameras, and an accelerometer glove. The accuracy of the synchronization and performance of the initial processing proved to be suitable for human action acquisition and recognition. The system was used for the acquisition of an extensive database of sign language gestures. User feedback indicated the recording process to be efficient, which is also evaluated in the paper. The system is publicly available, both in the form of a standalone application as well as source code, and can be easily customized to any type of sensor setup.