HRC of intelligent assembly system based on multimodal gesture control

As a natural expression of the human body, gestures are widely used in robot control. However, most gesture control methods rely only on single-modal feature representation, which has certain instability. To this end, this paper proposes a human-robot collaboration (HRC) framework for intelligent assembly systems with multimodal gesture control. The framework aims to perform gesture recognition on RGB data and RGB-D data through convolutional neural networks (CNN) and combine gesture data from both modalities for application in robot control, enabling data fusion and feature sharing. Perform asynchronous integration of image acquisition and gesture detection with spatially and time-aligned RGB frames and RGB-D frames to ensure real-time detection of gestures, and speed and separation monitoring to ensure the safety of the collaboration process. To verify the effectiveness of this framework, it was applied to the assembly scene of the integrally shrouded blade-rotor system. Experiments show that the HRC of the intelligent assembly system with multimodal gesture control can better realize the human-robot collaborative assembly with the integrally shrouded blade and improve the intelligence of the assembly process.