Self-supervised non-rigid structure from motion with improved training of Wasserstein GANs

This study proposes a self-supervised method to reconstruct 3D limbic structures from 2D landmarks extracted from a single view. The loss of self-consistency can be reduced by performing a random orthogonal projection of the reconstructed 3D structure. Thus, the training process can be self-supervised by using geometric self-consistency in the reconstruction-projection-reconstruction process. The self-supervised network mainly consists of graph convolution and Transformer encoders. This network is called the SS-Graphformer. By adding a discriminator, the SS-Graphformer is used as a generator to form a Wasserstein Generative Adversarial Network architecture with a Gradient Penalty to improve the accuracy of the reconstruction. It is experimentally demonstrated that the addition of the 2D structure discriminator can significantly improve the accuracy of the reconstruction.