Unified Visual-Semantic Embeddings: Bridging Vision and Language with Structured Meaning Representations

We propose the Unified Visual-Semantic Embeddings (Unified VSE) for learningajoint space of visual representation and textual semantics. The model unifies the embeddings of concepts at different levels: objects, attributes,relations, andfull scenes. We view the sentential semantics as a combination of different semantic components such as objects and relations; their embeddings are aligned with different image regions. A contrastive learningapproachis proposedfor the effective learning of this fine-grained alignmentfrom only image-captionpairs. We also present a simple yet effective approach that enforces the coverage of caption embeddings on the semantic components that appearin the sentence. We demonstrate that the Unified VSE outperforms baselines on cross-modal retrievaltasks; the enforcement of the semantic coverage improves the model's robustness in defending text-domain adversarialattacks. Moreover our model empowers the use of visual cues to accurately resolve word dependencies in novel sentences.