Aiming at the shortcomings of the existing color sorter machine for crop sorting, such as slow processing speed, low accuracy, and the dependence on experience value, a granular crop integrity identification algorithm based on convolutional neural network was proposed. Taking the classification of intact peanuts, skin damaged peanuts and half peanuts as instance, the three types of peanut images were acquired. After comparing the filtering effects of mean filtering, median filtering and Gaussian filtering, median filtering was adopted for image preprocessing. 407 effective peanut images were divided into the above three categories and manually labeled. Then the images were divided into training sets and validation sets, and the above three types of peanut pictures in the training set and the validation set were evenly distributed. A convolutional neural network with 4 convolutional layers, 4 pooling layers and 3 fully connected layers was built to extract the peanut image features. The accuracy of testing peanut classification on the CPU(central processing unit) platform combined GPU(graphics processing unit) was 90.91%. In contrast, the classification accuracy of the traditional BP neural network was 85.45%. It could be seen that the convolutional neural network algorithm constructed in this paper effectively improved the accuracy of granular crop recognition. In order to further improve the accuracy and real-time performance of the classification algorithm, it was necessary to optimize the established convolutional neural network. Over-fitting referred to the fact that when a model was overly complex, it could "memorize" the portion of random noise in each training data and forgot to "learn" the tendencyof the training data. In this paper, the regularization method was used to reduce the over-fitting, and the experimental results of L1 regularization and L2 regularization were compared. It was proved that the L2 regularization on the data set effectively improved the classification accuracy and reduced the over-fitting. In the process of training, the neural network used the back propagation algorithm, namely gradient descent and chain derivation rule, to optimize the neural network. The learning rate was an important parameter in the gradient descent algorithm. In this paper, the exponential decay method was used to set the learning rate. Firstly, a large learning rate was used to quickly obtain a better solution. Then, as the iteration continued, the learning rate was gradually reduced, making the model more stable in the later stage of training. The accuracy increase was larger, the latter was smaller, and the overall improvement was better than that before optimization, and the expected effect was achieved. In this paper, the moving average model was used to reduce the influence of noise in the training data on the model, and the training convergence speed was accelerated. The experiment proved that the accuracy fluctuation was reduced and the model stability was enhanced. Since the algorithm needed to be applied to the color sorting system, real-time judgment and processing of the materials on the conveyor belt required high real-time performance. Considering that the image information of peanut was relatively simple, the network structure could be simplified to improve the real-time performance. The simplified convolutional neural network consisted of 2 convolutional layers, 2 pooling layers, and 2 fully connected layers. The final optimization scheme included L2 norm regularization, exponential decay learning rate, moving average model and simplified network structure. The accuracy of optimized classification algorithm applied on the peanut data set was 98.18%, and the average processing time for detecting one peanut image was 18.3 ms, which demonstrated that the optimized convolutional neural network significantly improved the classification accuracy and real-time performance. The research work in this paper showed that the application of deep learning in the crop sorting field was feasible and effective. © 2018, Editorial Department of the Transactions of the Chinese Society of Agricultural Engineering. All right reserved.