Automatic image annotation using model fusion and multi-label selection algorithm

Automatic Image Annotation (AIA) aims to provide a semantic description for the content of image by assigning a set of textual labels. The recent approaches mainly focus on the improvement of single model and neglect the potential advantages of different models. In order to make full use of the advantages of different annotation models, Dual Model based on Multi-Label Selection Algorithm(DM-SA) is proposed in this research which combines a discriminative model with a nearest-neighbor-based model. The algorithm takes consideration of the advantages of each model, thus provides better annotation performance. A deep Convolutional Neural Network (CNN) is used to obtain visual representation of images first, then a discriminative model, CNN with Label Smoothing (CNN-LS), and a nearest-neighbor-based model, 2PKNN with Canonical Correlation Analysis (2PKNN-CCA) generate candidate label set respectively. Finally, a multi-label selection algorithm based on inverse document frequency is adopted to assign the final labels from two candidate label sets. Experimental results based on Corel5K and IAPRTC-12 datasets show that the proposed method can achieve state-of-the-art performance for average recall, 0.52 and 0.42 on Corel5K and IAPRTC-12 respectively.