Multimodal Fusion Representation Learning Based on Differential Privacy

Multimodal data for a certain target can often play a complementary role in information integration, but the diversification of the modal brings difficulties to the training of the model. Further, previous differential privacy works are only performed on a single modality. To tackle the problem, we choose deep representation learning to map different modalities data into the same subspace. This method of fusing multiple modalities uses low-rank decomposition based on Canonical Polyadic (CP) decomposition to implicitly obtain a high-dimensional tensor rich in mutual fusion information between multiple modalities, but explicitly obtain a low-dimensional representation. The perturbation that satisfies differential privacy is then carried out in the dimensional subspace. Experimental results show that it satisfies the data utility requirement while remaining suited privacy guarantee.