CDZL: a controllable diversity zero-shot image caption model using large language models

Zero-shot image caption generation enables machines to produce descriptions without meticulously curated training data. Addressing the issues present in current zero-shot image caption models, such as slow speed and low-quality captions, this paper proposes a controllable and diverse zero-shot image caption generation model based on a large model (Controlled Diverse Zero-Shot Image Captioning Model, CDZL). The CDZL model does not update parameters; instead, it uses an iterative approach to merge the predicted distributions of target words from various models to generate diverse image captions. By adding control signals, it can produce image captions with controllability. Leveraging the knowledge of large language models, CDZL makes the generated captions more aligned with the images. During the iterative process, the model incorporates the Metropolis-Hastings sampling idea, reducing the number of iterations by rejecting samples with excessively low scores, thereby accelerating the generation speed. Experimental results show that our method outperforms current state-of-the-art (SOTA) methods in most evaluation metrics and exhibits faster generation speed.