Crop Disease Diagnosis with Deep Learning-Based Image Captioning and Object Detection

The number of people participating in urban farming and its market size have been increasing recently. However, the technologies that assist the novice farmers are still limited. There are several previously researched deep learning-based crop disease diagnosis solutions. However, these techniques only focus on CNN-based disease detection and do not explain the characteristics of disease symptoms based on severity. In order to prevent the spread of diseases in crops, it is important to identify the characteristics of these disease symptoms in advance and cope with them as soon as possible. Therefore, we propose an improved crop disease diagnosis solution which can give practical help to novice farmers. The proposed solution consists of two representative deep learning-based methods: Image Captioning and Object Detection. The Image Captioning model describes prominent symptoms of the disease, according to severity in detail, by generating diagnostic sentences which are grammatically correct and semantically comprehensible, along with presenting the accurate name of it. Meanwhile, the Object Detection model detects the infected area to help farmers recognize which part is damaged and assure them of the accuracy of the diagnosis sentence generated by the Image Captioning model. The Image Captioning model in the proposed solution employs the InceptionV3 model as an encoder and the Transformer model as a decoder, while the Object Detection model of the proposed solution employs the YOLOv5 model. The average BLEU score of the Image Captioning model is 64.96%, which can be considered to have high performance of sentence generation and, meanwhile, the mAP50 for the Object Detection model is 0.382, which requires further improvement. Those results indicate that the proposed solution allows the precise and elaborate information of the crop diseases, thereby increasing the overall reliability of the diagnosis.