Dynamic text prompt joint multimodal features for accurate plant disease image captioning

Plant disease captioning is crucial for agricultural pest and disease prevention. However, generating accurate captions for plant disease images remains challenging because of the limited availability of datasets and complex disease manifestations. In this study, we propose BLIP-DP, a novel approach that dynamically generates prompts to enhance multimodal feature fusion for plant disease image captioning. We manually annotated over 20,000 images from the PlantVillage dataset and developed a model that incorporates a ViT, BERT, and a visual question-answering module. The experimental results demonstrate the effectiveness of our method, achieving a BLEU-4 score of 83.4, which represents an improvement of 4.1 over the original baseline, outperforming previous methods. Our work not only improves the accuracy of plant disease descriptions but also paves the way for future research in this underexplored area. The code is available at https://github.com/zilonghh/BLIP-DP.