Improving label quality in crowdsourcing using deep co-teaching-based noise correction

In the crowdsourcing scenario, repeated labeling is employed to obtain each instance’s multiple noisy label set from different crowd workers on the Internet, and then a ground truth inference method is used to obtain its integrated label. However, no matter which ground truth inference method is used, a certain level of noise remains in the integrated labels. To improve the quality of the integrated labels, a number of noise correction methods have been proposed in recent years. However, to the best of our knowledge, almost all these methods filter out too many instances that are regarded as noise instances and thus can only use a few clean instances to learn classifiers for noise correction. In this paper, we propose a two-stage noise correction method called deep co-teaching-based noise correction (DCTNC), which can learn not only from clean instances but also from noise instances adaptively. In the first stage, original instances are split into a clean set and a noise set according to the confusion level of their multiple noisy label sets. In the second stage, we at first train a deep network on the clean set and then use it to guide the training of another two deep networks on the noise set through an improved co-teaching algorithm. Finally, we use the trained three deep networks to correct the instances in the noise set. The experimental results on eleven simulated datasets and one real-world dataset show that the proposed DCTNC achieves new state-of-the-art results.