Robust multimodal federated learning for incomplete modalities

Consumer electronics are continuously collecting multimodal data, such as audio, video, and so on. A multimodal learning mechanism can be adopted to deal with these data. Due to the consideration of privacy protection, some successful attempts at multimodal federated learning (MMFed) have been conducted. However, real-world multimodal data is usually missing modalities, which can significantly affect the accuracy of the global model in MMFed. Effectively fusing and analyzing multimodal data with incompleteness remains a challenging problem. To tackle this problem, we propose a robust Multimodal Federated Learning Framework for Incomplete Modalities (FedInMM). More specifically, we design a Long Short-Term Memory (LSTM)-based module to extract the information in the temporal sequence. We dynamically learn a weight map to rescale the feature in each modality and formulate the different contributions of features. And then the content of each modality is further fused to form a uniform representation of all modalities of data. By considering the temporal dependency and intra-relation of multi-modalities automatically through the learning stage, this MMFed framework can efficiently mitigate the effects of missing modalities. By using two multimodal datasets, DEAP and AReM, we have conducted comprehensive experiments by simulating different levels of incompleteness. Experimental results demonstrate that FedInMM outperforms other approaches and can train highly accurate models on datasets comprising different incompleteness patterns, which is more appropriate for integration into a practical multimodal application.