Cross-Modal Integration and Transfer Learning Using Fuzzy Logic Techniques for Intelligent Upper Limb Prosthesis

The integration and interaction of proprioception and exteroception in the human multisensory network facilitate high-level cognitive functionalities, such as cross-modal integration, recognition, and imagination for accurate evaluation and comprehensive understanding of the multimodal world. In this article, we propose a novel cross-modal integration framework for the upper limb prosthesis based on type-2 fuzzy logic system (FLS), which can facilitate the high-level cognitive and dexterous manipulation of the prosthesis by combing human's surface electromyography (sEMG) with the computer vision. First, a transfer learning approach is proposed to improve the decoding of human's intent and enhance the effectiveness of skill transition. Then the sEMG signals and image information are integrated to jointly determine the grasp posture of the bionic hand based on fuzzy decision strategy. Fusing multisensory data and using cross-modal integration, the system is capable of crossmodally recognizing multimodal information. In order to realize the prosthesis automatically reaching the target position under the guidance of computer vision, an interval type-2 fuzzy logic controller considering uncertain dynamic parameters and disturbance is designed. Experiments are performed in some typical 3C assembly scenarios, and results show that our proposed strategy can obviously improve the accuracy of grasp posture selection and trajectory tracking effect, which brings more potential job opportunities with hope to amputees, and provides a promising approach toward robotic sensing and perception.