Multimodal Medical Imaging Using Modern Deep Learning Approaches

Multimodal medical imaging is gaining prominence in clinical practice as well as in research studies. Multimodal image analysis (MIA) in conjunction with ensemble learning strategies gave rise to explosion in popularity and adding special benefits for medical-related applications. Inspired by recent successes of deep learning techniques in medical imaging, we design an algorithmic structure that enables supervised MIA with Cross-Modality Fusion at preprocessing stage, the classifier level as well as the decision-making step. Using deep convolutional neural networks, we proposed an algorithm for image segmentation to determine the lesions caused by soft tissue tumors. This is done using multimodal images by MRI tomography as well as PET. The NN built with multimodal images performs better than networks built with single-modal images. In the case of tumor segmentation, an image that is fused within the neural network (i.e., fused within the convolutional layer or totally joined layers) is more effective as compared to using pictures that fuse the network's output. This work offers specific recommendations for the development and application of MIA.