TOPIC MODEL AND SIMILARITY CALCULATION OF TEXT ON SPARK

Since the 21st century, with the popularization and development of the Internet, the volume of text data on the Internet has been increasing rapidly. To extract the theme of people, need from text content, at the same time in order to solve problems such as excessive dimension of vector in model and dependence of word frequency, David M. Blei put forward a helpful and efficient method based on topic model, which also known as implicit Dirichlet distribution (Latent Dirichlet Allocation). The LDA model uses the Gibbs sampling method to estimate the parameters of the document-topic, topic-word distribution pattern, and express the text content as the probability model of the subject model to calculate the similarity of the document. In this paper, Spark as a LDA algorithm to run the computing platform, the use of Spark parallel computing advantages of a large number of text corpus processing, and on the basis of LDA proposed a document similarity calculation method.