Three-dimensional virtual try-on network based on attention mechanism and vision transformer

Objective Three-dimensional (3-D) virtual try-on can provide an intuitive and realistic view for online shopping and has great potential commercial value. However, there are some problems in the existing 3-D virtual try-on network, such as inaccurate generated 3-D human models, unclear model edges and excessive clothing deformation in the virtual fitting, which greatly limit the application of this technology in real scenarios. Method In order to solve the above problems, this research proposed the network named T3D-VTON, a deep neural network introducing convolutional attention mechanism and vision transformer. The network was designed to have three modules: 1) a convolutional block attention module that was added to the feature extraction module to make the network focus on the key information and reduce the influence of irrelevant information; 2) a depth estimation network which was created to adopt an encoder-decoder structure for the establishment of a multiscale neural network combining Resnet and transformer; 3) a feature fusion module that aimed to fuse 2-D and 3-D information to obtain the final 3-D virtual fitting model. The effect of adding the convolution attention mechanism and vision transformer module on the performance of the network was investigated in details. The performance of the network is mainly expressed by the virtual fitting results and the accuracy of the human body model. Qualitative and quantitative comparative analyses were conducted between this experiment and the benchmark network. Results The quantitative experimental results showed that the structure similarity index measure (SSIM) was improved by 0.015 7 compared with the baseline network, and the peak signal-to-noise ratio (PSNR) improved by 0.113 2. The above results indicated that the image generation quality is improved without much loss of information. In terms of human model generation accuracy, compared to the baseline network, absolute relative error was reduced by 0.037 and square relative error was reduced by 0.014 in the results of depth estimation, indicating that the 3-D human model generated by this network was more accurate and the depth in the depth map was more consistent with the original given ground truth. The qualitative experimental results showed that the deformation of the garment fitted the original area of the target body more closely without excessive deformation and reduced the generation of garment artifacts. When dealing with complex textures, the network was able to better preserve the pattern and material of the garment fabric. The generated 3-D human body try-on model showed the front and side effects of the human body model, suggesting that the 3-D human body model generated by the network presents clearer contour edges and effectively eliminates the adhesion between the arms and the abdomen. When the knees are close to each other for example, the network would be able to eliminate the adhesion between the knees. Conclusion The convolutional block attention module and vision transformer introduced by the T3D-VTON network are able to preserve the textural patterns and brand logos on the garment surface when dealing with complex textures. The structure can effectively regulate the garment deformation and blend reasonably with the dressing area of the target character. When generating the 3-D human body model, the network can produce a clearer edge and has more accurate shape generation capability. The method can finally present a 3-D human body model with richer surface texture and more accurate body shape, which provides a fast and economical solution to realize a single image to 3-D virtual application. © 2023 China Textile Engineering Society. All rights reserved.