PAEE: Parameter-Efficient and Data-Effective Image Captioning Model with Knowledge Prompter and Cross-Modal Representation Aligner

Large-scale pre-trained models and research on massive data have achieved state-of-the-art results in image captioning technology. However, the high cost of pre-training and fine-tuning has become a significant issue that needs to be considered. In this paper, we propose PAEE, a parameter-efficient and data-effective image captioning model that generates captions based on the input image encoding and the knowledge obtained from the newly introduced Knowledge Prompter. In PAEE, the only module that needs to be learned is the Cross-modal Representation Aligner (CRA) introduced between the visual encoder and language decoder, which facilitates the language model's better adaptation to visual representation. The entire model greatly reduces the cost of pre-training and fine-tuning. Extensive experiments demonstrate that PAEE maintains competitive performance compared to large-scale pre-trained models and similar approaches, while reducing the number of trainable parameters. We design two new datasets to explore the data utilization ability of PAEE and discover that it can effectively use new data and achieve domain transfer without any training or fine-tuning. Additionally, we introduce the concept of small-data learning and find that PAEE has data-effective characteristics in limited computing resources and performs well even with fewer training samples.