DSAGAN: A generative adversarial network based on dual-stream attention mechanism for anatomical and functional image fusion

In recent years, extensive multimodal medical image fusion algorithms have been pro-posed. However, existing methods are primarily based on specific transformation theories. There are many problems with existing algorithms, such as poor adaptability, low effi-ciency and blurry details. To address these problems, this paper proposes a generative adversarial network based on dual-stream attention mechanism (DSAGAN) for anatomical and functional image fusion. The dual-stream architecture and multiscale convolutions are utilized to extract deep features. In addition, the attention mechanism is utilized to further enhance the fused features. Then, the fusion images and multimodal input images are put into the discriminator. In the update stage of the discriminator, we expect to judge the multimodal images as real, and to judge the fusion images as fake. Furthermore, the fusion images are expected to be judged as real in the update stage of the generator, forcing the generator to improve the fusion quality. The training process continues until the generator and discriminator reach a Nash equilibrium. After training, the fusion images can be obtained directly after inputting anatomical and functional images. Compared with the ref-erence algorithms, DSAGAN consumes less fusion time and achieves better objective met-rics in terms of QAG, QEN and QNIQE. (c) 2021 Elsevier Inc. All rights reserved.