From single- to multi-modal remote sensing imagery interpretation: a survey and taxonomy

Modality is a source or form of information. Through various modal information, humans can perceive the world from multiple perspectives. Simultaneously, the observation of remote sensing (RS) is multimodal. We observe the world macroscopically through panchromatic, Lidar, and other modal sensors. Multimodal observation of remote sensing has become an active area, which is beneficial for urban planning, monitoring, and other applications. Despite numerous advancements in this area, there has still not been a comprehensive assessment that provides a systematic overview with a unified evaluation. Accordingly, in this survey paper, we first highlight the key differences between single- and multimodal RS imagery interpretation, then use these differences to guide our research survey of multimodal RS imagery interpretation in a cascaded structure. Finally, some potential future research directions are explored and outlined. We hope that this survey will serve as a starting point for researchers to review state-of-the-art developments and work on multimodal research.