Weakly supervised grounded image captioning with semantic matching

Visual attention has been extensively adopted in many tasks, such as image captioning. It not only improves the performance of image captioning but is also used to enhance the quality of caption rationality. Rationality can be understood as the ability to maintain attention on the correct regions while generating words or phrases. This is critical for alleviating the problems of object hallucinations. Recently, many researchers have devoted to improving grounding accuracy by linking generated object words or phrases to appropriate regions of the image. However, collecting word-region alignment is expensive and limited, and the generated object words may not appear in the annotation sentences. To address this challenge, we propose a weakly supervised grounded image captioning method. Specifically, we design a region-word matching block to estimate the match scores for the candidate nouns with all regions. Compared to manual annotations, the match score may contain some mistakes. To make the captioning model compatible with these mistakes, we design a reinforcement loss that takes into account both attention weights and match scores. This allows the captioning model to generate a more accurate and grounded sentence. Experimental results on two commonly used benchmark datasets (MSCOCO, Flickr30k) demonstrate the superiority of the proposed blocks. Extensive ablation studies also validate the effectiveness and robustness of the proposed modules. Last but not least, our blocks are available in a variety of captioning models and do not require additional label or extra time consumption in inference stage.