In industrial production, medical gloves with tear, stain and other defects will be produced. In traditional manual mode, the efficiency and accuracy of defect detection depend on the proficiency of spot-check workers, which results in uneven glove product quality. In this paper, a surface defect detection system of medical gloves based on deep learning is designed for the automatic detection with high efficiency and accuracy. According to the industrial requirements of high real-time, the system adopts a cache scheme to improve the data reading and writing speed, and an Open Neural Network Exchange (ONNX) to effectively improve the speed of model reasoning. For the demands of high detection accuracy, the system designs a dual model detection strategy, which divides texture detection and edge detection into two steps. The advantage of this strategy is to remove most useless information while ensuring the effective information of the image. Furthermore, two auxiliary models are used to promote the accuracy of detection based on classification methods. Finally, experiments are proposed to verify the functional indicators of the system. After the on-site test of the production line in the medical glove factory, the system has the ability to detect the gloves of two production lines with high real-time. The product missed detection rate is less than 2%, and the product mistakenly picked rate is less than 5/10000. Verified by the industry of gloves, the system can be put into production line.