Segmentation-based Unsupervised Phrase Detection

In this paper, we propose a new approach to unsupervised phrase detection that is based on a sentence segmentation. Unlike the existing approach that examines only word-based statistics, the proposed method detects phrases by considering the most likely segmentation for each sentence. We develop a Bayesian model that estimates phrase boundaries and the grammatical roles of each phrase at the same time, which can be trained in an unsupervised manner by using Gibbs sampling. The experimental results show that the phrase detection by using the proposed model can recognize about 30 times more phrases than the existing popular method in the same precision because of the successful detection of infrequent phrases.