Cross-Modal Generation of Tactile Friction Coefficient From Audio and Visual Measurements by Transformer

Generating tactile data (e.g., friction coefficient) from audio and visual modalities can avoid time-consuming practical measurements and ensure high-fidelity haptic rendering of surface textures. In this article, we present a Transformer-based method for cross-modal generation of the tactile friction coefficient. Using the self-attention mechanism, we jointly encode the amplitude and phase information of audio spectrums and RGB images to extract global and local features. Then, we convert the joint coding features into tactile decoding features using a Transformer module in a bottleneck converter. We continuously decode and reconstruct them to obtain amplitude and phase information of tactile friction coefficients. Finally, we convert this information into 1-D friction coefficients using inverse short-time Fourier transform (ISTFT). Evaluations of the LMT Haptic Material Database confirm the obvious performance improvement of the proposed method. Furthermore, with the generated friction by the Transformer and the custom-designed electrovibration device, a novel rendering method is proposed, which simultaneously utilizes the amplitude and frequency of driving signals to display tactile textures on touchscreens. User experiments are organized to evaluate the rendering fidelity of the generated friction coefficient. The two-way analysis of variance (ANOVA) with repeated measurements indicates that the rendering fidelity of the Transformer method is significantly improved compared with the contrast methods ( p < 0.05).