A Multimodal Federated Learning Framework for Modality Incomplete Scenarios in Healthcare

Multimodal federated learning has been found extensive application in healthcare for collaborative model training while ensuring data privacy and security. However, most existing methods assume completeness of all modalities on each client, disregarding the issue of modality incongruity between clients due to missing modalities. In this study, we propose a novel multimodal federated learning framework to address knowledge sharing and collaborative learning among heterogeneous clients in scenarios with incomplete modalities. To tackle traditional federated aggregation failures resulting from incomplete modalities, our approach initially employs a cluster stepwise aggregation strategy to group clients with consistent modalities into clusters, facilitating inter-cluster communication by aggregating feature encoders of the same modality. Furthermore, to compensate for the insufficient ability of global gradient information in conveying semantic features of heterogeneous clients, a prototype contrastive integration strategy is introduced to achieve effective semantic knowledge sharing through intra- and inter-modality prototype contrastive learning. Experimental results demonstrate the effectiveness of our approach in scenarios with incomplete modalities.