Computer vision detection of foreign objects in walnuts using deep learning

Detection of foreign objects is crucial for quantitative image analysis in numerous food quality and safety inspection applications. Rapid detection of foreign objects in walnuts using computer vision still faces challenge due to the irregular shapes and complex features of foreign objects. Some detection methods require application specific transforms, expertly designed constraints and model parameters, and have limited detection performance due to their maintenance costs. In recent years, deep learning has become a focus in different research fields, because methods based on deep learning are able to directly learn features from training data. In this study, we apply two different convolutional neural network structures to walnut images for automatically segmenting images and detecting different-sized natural foreign objects (e.g., flesh leaf debris, dried leaf debris and gravel dust) and man-made foreign objects (e.g., paper scraps, packing material, plastic scraps and metal parts). The proposed deep-learning method is simpler because it avoids extracting features manually, and overcomes the conglomeration phenomenon between walnuts and foreign objects in actual images. The proposed method is able to correctly segment 99.5% of the object regions in the 101 test images and to correctly classify 95% of the foreign objects in the 277 validation images. The segmentation and detection processing time of each image was less than 50 ms. Future work will focus on deep learning using multi-waveband imaging hardware and fast on-line inspection control for the equipment and robots.