Visual sentiment analysis using data-augmented deep transfer learning techniques

There has been a growing trend among users of social media platforms to express their emotions using visual content. Visual sentiment analysis is the process of understanding the emotional polarity of images or videos and is still considered a challenging problem in artificial intelligence. Most of the existing models are based on robust machine learning or deep learning techniques. The idea of using deep transfer learning techniques for visual sentiment analysis is fairly new. In this paper, we propose a new approach using data-augmented-transfer learning architecture consisting of a pre-trained VGG16 model that is fine-tuned using SVM with augmented training data. For fine-tuning and evaluation, we initially use two Twitter image datasets. We further validated the proposed model on a third dataset. The commonly used geometric augmentation methods such as rotation, zoom range, width shift, height shift, shear range and horizontal flip were are used. We compare our proposed VGG16-SVM model with 3 other state-of-the-art deep models commonly used for transfer learning and 4 machine learning models (besides SVM) used for fine-tuning. The results show that VGG16-SVM produces the overall best accuracy (94%) and recall (96%) among all transfer learning and machine learning pairs. We also show that our proposed model outperforms all previous studies that use the same dataset.