Multimodal federated learning: Concept, methods, applications and future directions

Multimodal learning mines and analyzes multimodal data in reality to better understand and appreciate the world around people. However, how to exploit this rich multimodal data without violating user privacy is a key issue. Federated learning is a privacy-conscious alternative to centralized machine learning, therefore many researchers have combined federated learning with multimodal learning to break down data barriers for the purpose of jointly leveraging multiple modal data from different clients for modeling. In order to provide a systematic summarize of multimodal federated learning, this paper describes the basic mode of multimodal federated learning, multimodal fusion based on federated learning, multimodal federated learning optimization and multimodal federated learning application, and introduces each type of multimodal federated learning methods in detail. Finally, the future research trends of multimodal federated learning are discussed and analyzed, mainly including the optimization of multimodal federated learning, privacy- preserving techniques for multimodal federated learning, multimodal federated few-shot learning & multimodal federated semi-supervised learning, and data and knowledge-driven multimodal federated learning.