TVT-Transformer: A Tactile-visual-textual fusion network for object recognition

In the pursuit of higher levels of intelligence, embodied intelligences need to integrate information from multiple perceptual channels through a multimodal information fusion mechanism to comprehensively understand the surrounding scene and the manipulated objects. Most of the current multimodal perception research focuses on the fusion of vision-based, and most of them involve only the fusion of two modalities, while the fusion of three or more modalities is seldom explored. Therefore, we propose TVT-Transformer (TVT: Tactile-Visual-Textual): a new framework for joint learning from three modal data: tactile, visual and semantic text. The approach utilizes Attention mechanism to deeply mine and align information from different perceptual modalities, allowing the model to efficiently integrate features from tactile, visual, and semantic text data, enabling a deeper level of information fusion through cross-modal interaction. The framework introduces a novel semantic representation approach to generate standardized semantic descriptions by combining human observation and touch interactions with objects. Next, the semantic descriptions are encoded using a pre-trained Bert model, aligning them with visual and tactile information. Following this, the Query (Q), Key (K), and Value (V) components from tactile, visual, and textual modalities are integrated into a unified Q, K, and V. Subsequently, the Attention mechanism within the Transformer architecture is employed for cross-attention computation, enabling more accurate and efficient cross-modal feature integration and understanding. The textual modality provides semantic support that enhances the effectiveness of information integration and improves the accuracy of object recognition. With tactile sensor data, visual image data, and corresponding semantic text descriptions as input data, the method has been validated for its effectiveness and superiority on both publicly available and self-made datasets, enhances the feature expression capability, achieves significant performance enhancement in multimodal data integration, and improves the accuracy of object recognition. Compared with the classical Transformer, TVT-Transformer is able to effectively fuse visual, tactile, and semantic textual information through a cross-modal Self-Attention mechanism, and exhibits greater adaptability and robustness in processing multimodal information. This study not only provides a new perspective in the field of multimodal information fusion, but also provides a strong technical support for the development of embodied intelligence. The TVT-Transformer framework has a wide range of potential applications, and it is expected to play an important role in the future in the fields of intelligent robotics, human-robot interaction, and assisted decision-making. The resource has been released at https://github.com/huakaichengbei/MSDO/tree/master.