3D mesh transformer: A hierarchical neural network with local shape tokens

Self-attention networks have revolutionized Natural Language Processing (NLP) and are making impres-sive strides in image analysis tasks such as image classification and object detection. Inspired by this suc-cess, we specifically design a novel self-attention mechanism between local shapes and build a shape Transformer. We split the 3D mesh model into shape patches, which we call shape tokens, and provide polynomial fitting representations of these patches as input to the shape Transformer. The shape token encodes local geometric information and resembles the token (word) status in NLP. The simplification of the mesh model provides a hierarchical multiresolution structure, which allows us to realize the fea-ture learning of a multilayer Transformer. We set high-level features formed by the shape Transformer as visual tokens and propose a vector-type self-attention mechanism to construct a 3D visual Transformer. Finally, we realized a hierarchical network structure based on local shape tokens and high-level visual tokens. Experiments show that our fusion network of 3D shape Transformer with explicit local shape con-text augmentation and 3D visual Transformer with multi-level structural feature learning achieves excel-lent performance on shape classification and part segmentation tasks.(c) 2022 Elsevier B.V. All rights reserved.