Statistical quality estimation for partially subjective classification tasks through crowdsourcing

When constructing a large-scale data resource, the quality of artifacts has great significance, especially when they are generated by creators through crowdsourcing. A widely used approach is to estimate the quality of each artifact based on evaluations by reviewers. However, the commonly used vote-counting method to aggregate reviewers’ evaluations does not work effectively for partially subjective tasks. In such a task, a single correct answer cannot necessarily be defined. We propose a statistical quality estimation method for partially subjective classification tasks to infer the quality of artifacts considering the abilities and biases of creators and reviewers as latent variables. In our experiments, we use the partially subjective task of classifying speech into one of the following four attitudes: agreement, disagreement, stalling, and question. We collect a speech corpus through crowdsourcing and apply the proposed method to it. The results show that the proposed method estimates the quality of speech more effectively than vote aggregation, as measured by correlation with a fine-grained classification performed by experts. Furthermore, we compare the speech attitude classification performance of a neural network model on two subsets of our corpus extracted using the voting and proposed methods. The results indicate that we can effectively extract a consistent and high-quality subset of a corpus using the proposed method. This method facilitates the efficient collection of large-scale data resources for mutually exclusive classification, even if the task is partially subjective.