Disentangle, Align and Fuse for Multimodal and Semi-Supervised Image Segmentation

被引:48
作者
Chartsias, Agisilaos [1 ]
Papanastasiou, Giorgos [2 ,3 ]
Wang, Chengjia [2 ,3 ]
Semple, Scott [2 ,3 ]
Newby, David E. [2 ,3 ]
Dharmakumar, Rohan [4 ]
Tsaftaris, Sotirios A. [1 ,5 ]
机构
[1] Univ Edinburgh, Sch Engn, Edinburgh EH9 3JL, Midlothian, Scotland
[2] QMRI, Edinburgh Imaging Facil, Edinburgh EH16 4TJ, Midlothian, Scotland
[3] Ctr Cardiovasc Sci, Edinburgh EH16 4TJ, Midlothian, Scotland
[4] Cedars Sinai Med Ctr, Los Angeles, CA 90048 USA
[5] Alan Turing Inst, London NW1 2DB, England
基金
英国医学研究理事会; 美国国家卫生研究院; 英国工程与自然科学研究理事会;
关键词
Image segmentation; Biomedical imaging; Annotations; Training; Semantics; Decoding; Multimodal segmentation; disentanglement; magnetic resonance imaging; MAGNETIC-RESONANCE; MRI; SEQUENCE; NETWORK; SHAPE;
D O I
10.1109/TMI.2020.3036584
中图分类号
TP39 [计算机的应用];
学科分类号
081203 ; 0835 ;
摘要
Magnetic resonance (MR) protocols rely on several sequences to assess pathology and organ status properly. Despite advances in image analysis, we tend to treat each sequence, here termed modality, in isolation. Taking advantage of the common information shared between modalities (an organ's anatomy) is beneficial for multi-modality processing and learning. However, we must overcome inherent anatomical misregistrations and disparities in signal intensity across the modalities to obtain this benefit. We present a method that offers improved segmentation accuracy of the modality of interest (over a single input model), by learning to leverage information present in other modalities, even if few (semi-supervised) or no (unsupervised) annotations are available for this specific modality. Core to our method is learning a disentangled decomposition into anatomical and imaging factors. Shared anatomical factors from the different inputs are jointly processed and fused to extract more accurate segmentation masks. Image misregistrations are corrected with a Spatial Transformer Network, which non-linearly aligns the anatomical factors. The imaging factor captures signal intensity characteristics across different modality data and is used for image reconstruction, enabling semi-supervised learning. Temporal and slice pairing between inputs are learned dynamically. We demonstrate applications in Late Gadolinium Enhanced (LGE) and Blood Oxygenation Level Dependent (BOLD) cardiac segmentation, as well as in T2 abdominal segmentation. Code is available at https://github.com/vios-s/multimodal_segmentation.
引用
收藏
页码:781 / 792
页数:12
相关论文
共 54 条
[1]  
Alemi A.A., 2016, PROC INT C LEARN REP
[2]  
[Anonymous], 2019, PROC ICLR SOCIAL GOO
[3]  
Burgess C. P., 2017, P WORKSH LEARN DIS R, P1
[4]   Combining Multi-Sequence and Synthetic Images for Improved Segmentation of Late Gadolinium Enhancement Cardiac MRI [J].
Campello, Victor M. ;
Martin-Isla, Carlos ;
Izquierdo, Cristian ;
Petersen, Steffen E. ;
Gonzalez Ballester, Miguel A. ;
Lekadir, Karim .
STATISTICAL ATLASES AND COMPUTATIONAL MODELS OF THE HEART: MULTI-SEQUENCE CMR SEGMENTATION, CRT-EPIGGY AND LV FULL QUANTIFICATION CHALLENGES, 2020, 12009 :290-299
[5]   Multimodal Cardiac Segmentation Using Disentangled Representation Learning [J].
Chartsias, Agisilaos ;
Papanastasiou, Giorgos ;
Wang, Chengjia ;
Stirrat, Colin ;
Semple, Scott ;
Newby, David ;
Dharmakumar, Rohan ;
Tsaftaris, Sotirios A. .
STATISTICAL ATLASES AND COMPUTATIONAL MODELS OF THE HEART: MULTI-SEQUENCE CMR SEGMENTATION, CRT-EPIGGY AND LV FULL QUANTIFICATION CHALLENGES, 2020, 12009 :128-137
[6]   Disentangled representation learning in cardiac image analysis [J].
Chartsias, Agisilaos ;
Joyce, Thomas ;
Papanastasiou, Giorgos ;
Semple, Scott ;
Williams, Michelle ;
Newby, David E. ;
Dharmakumar, Rohan ;
Tsaftaris, Sotirios A. .
MEDICAL IMAGE ANALYSIS, 2019, 58
[7]   Factorised Spatial Representation Learning: Application in Semi-supervised Myocardial Segmentation [J].
Chartsias, Agisilaos ;
Joyce, Thomas ;
Papanastasiou, Giorgos ;
Semple, Scott ;
Williams, Michelle ;
Newby, David ;
Dharmakumar, Rohan ;
Tsaftaris, Sotirios A. .
MEDICAL IMAGE COMPUTING AND COMPUTER ASSISTED INTERVENTION - MICCAI 2018, PT II, 2018, 11071 :490-498
[8]   Unsupervised Multi-modal Style Transfer for Cardiac MR Segmentation [J].
Chen, Chen ;
Ouyang, Cheng ;
Tarroni, Giacomo ;
Schlemper, Jo ;
Qiu, Huaqi ;
Bai, Wenjia ;
Rueckert, Daniel .
STATISTICAL ATLASES AND COMPUTATIONAL MODELS OF THE HEART: MULTI-SEQUENCE CMR SEGMENTATION, CRT-EPIGGY AND LV FULL QUANTIFICATION CHALLENGES, 2020, 12009 :209-219
[9]   Robust Multimodal Brain Tumor Segmentation via Feature Disentanglement and Gated Fusion [J].
Chen, Cheng ;
Dou, Qi ;
Jin, Yueming ;
Chen, Hao ;
Qin, Jing ;
Pheng-Ann Heng .
MEDICAL IMAGE COMPUTING AND COMPUTER ASSISTED INTERVENTION - MICCAI 2019, PT III, 2019, 11766 :447-456
[10]  
Chen C, 2019, AAAI CONF ARTIF INTE, P865