FUSION3D: Multimodal data fusion for 3D shape reconstruction: A soft-sensing approach

Traditional high-fidelity imaging techniques, such as X-ray computer tomography (CT), excel in capturing intricate shape details through high-resolution two-dimensional (2D) images. However, the extensive cost or impracticality of obtaining X-ray images poses a significant challenge in various applications (e.g., dense metal components in manufacturing). To overcome this limitation, we propose a soft sensing approach, named FUSION3D, aimed at reconstructing three-dimensional (3D) shapes from 2D sliced images and heterogenous process inputs using a novel model based on multi-modal process knowledge. The developed FUSION3D method is built upon a deep learning framework that utilizes continuous normalizing flow to model the complex shape distribution of 3D objects. We propose a principled probabilistic framework to regularize 3D point clouds by modeling them as a distribution of distributions. Specifically, we learn a two-level hierarchy of distributions where the first level is the distribution of shapes, and the second level is the distribution of points given a shape. Our generative regularization model learns each level of the distribution with a continuous normalizing flow. The invertibility of normalizing flows enables the computation of the likelihood during training and allows us to train our model in the variational inference framework. Our methodology is validated through a case study in additive manufacturing, demonstrating its effectiveness as a soft sensing approach for accurate 3D shape reconstruction without relying on high-fidelity imaging techniques like CT during model deployment.