Cross-media retrieval: Concepts, advances and challenges

Cross-media retrieval is an emerging comprehensive research topic, which seeks to provide more effective retrieval approach so that internet users could query multimedia objects by examples in the form of different media. For example, users can query images by submitting an example audio clip in a cross-media retrieval system and vice versa. Clearly, a cross-media retrieval system better fits for human habits and is thus more powerful in retrieval performance. In order to achieve cross-media retrieval, we need to resolve the problem of semantic understanding and mappings among heterogeneous low-level multi-modal features spaces, such as judging the correlation between visual contents and auditory contents in accordance with human perception. In this paper we give the concept of cross-media retrieval, and two effective approaches for the two kinds of cross-media retrieval, namely Correlation Isomorphic Space Learning (CISL) for media object retrieval and Manifold Semantic Space Learning (MSSL) for multimedia document retrieval. CISL uses canonical correlation analysis to map pairs of heterogeneous multi-modal features into an integrity semantic subspace where canonical correlations are furthest preserved. MSSL implements manifolds learning to explore the relationship among multimedia documents and media objects within them respectively. Experiment results are encouraging and indicate that the performance of the proposed approaches is effective.