POS Weighted TF-IDF Algorithm and its Application for an MOOC Search Engine

Term Frequency-Inverse Document Frequency (TF-IDF) has been one of the most highly used information retrieval methods for many years. Although there are several variants of TF-IDF optimizing for solving various problems, very few of them considered the properties of the query terms themselves. We found that there could be a big potential for improvement. When people type out a query, usually the verbs and the nouns are the primary keywords that directly define the query. The adjectives and adverbs are generally the secondary keywords, which describe the query more accurately. Other terms might not be as important as the terms just mentioned and could be the tertiary keywords. Based on this fact, this paper proposes an algorithm improved upon the original TF-IDF algorithm - POS Weighted TF-IDF algorithm. This algorithm takes every query term's part of speech (POS) into account and assigns each query term frequency a different weight value according to the POS of that term. Based on the POS Weighted TF-IDF Algorithm, we developed COURSES, a massive open online courses (MOOC) search engine, and achieved very positive results, which shows the effectiveness of the proposed algorithm.