Semantic Inference Network for Human-Object Interaction Detection

Recently many efforts have been made to understand the scenes in images. The interactions between human and objects are usually of great significance to scene understanding. In this paper, we focus on the task of detecting human-object interactions (HOI), which is to detect triplets < human, verb, object > in challenging daily images. We propose a novel model which introduces a semantic stream and a new form of loss function. Our intuition is that the semantic information of object classes is beneficial to HOI detection. Semantic information is extracted by embedding the category information of objects with pre-trained BERT model. On the other hand, we find that the HOI task suffers severely from extreme imbalance between positive and negative samples. We propose a weighted focal loss (WFL) to tackle this problem. The results show that our method achieves a gain of 5% compared with our baseline.