A text similarity measurement combining word semantic information with TF-IDF method

Traditional text similarity measurements use TF-IDF method to model text documents as term frequency vectors, and compute similarity between text documents by using cosine similarity. These methods ignore semantic information of text documents, and semantic information enhanced methods distinguish between text documents poorly because extended vectors with semantic similar terms aggravate the curse of dimensionality. This paper proposes a similarity measurement, which is based on TF-IDF method, and analyzes similarity between important terms in text documents. This approach uses NLP technology to pre-process text, and uses TF-IDF method to filter those key terms that have higher TF-IDF value than other common terms. With the proposed data structure TSWT (Term Similarity Weight Tree) and the definition of semantic similarity, this paper resolves the semantic information of those key terms to compute similarities between text documents. Finally, several K-Means clustering methods is used for evaluating performance of the new text document similarity. By comparing with TF-IDF and another the-state-of-art semantic information based similarity method, experimental results on benchmark corpus demonstrate that it can promote the evaluation metrics of F-Measure.