Soft Transfer Learning via Gradient Diagnosis for Visual Relationship Detection

Detecting all visual relationships (e.g. "person-wear-shirt") is posed as the most fundamental task towards the ultimate semantic reasoning. However, due to the rich context embedded in the image and diverse language ambiguities (e.g. person vs. man), it is unrealistic to annotate all possible relationships for providing a noise-free supervised setting. All prior approaches simply adopt the traditional fully-supervised detection pipeline and ignore the effect of incomplete annotations on model convergence, resulting in the unstable optimization and unsatisfactory performance. In this work, we make the first attempt to address this critical incomplete annotations issue and reformulate this task via the Soft Transfer Learning (STL), which aims to transfer knowledge learned from the annotations in hand into the uncertain pairs in a self-supervised way. The knowledge transfer process is inferred from a principled gradient diagnosis. Extensive experiments on VRD and the large-scale VG benchmarks demonstrate the superiority of our STL method.