A new category-based weighting scheme for automated text categorization

Term weighting is a strategy that assigns weights to terms in order to improve the performance of text categorization. In this paper, we propose a new category-based term weighting scheme named the probability of relevance frequency (prf), which uses available labeling information to assign appropriate weights to terms. The main idea of prf is that the more concentrated a highfrequency term is in the positive category than in the negative category, the more contribution it makes in separating the positive samples from the negative samples. By replacing word features with category-based features, the dimensionality of the document feature space can be reduced from tens of thousands to a small number of categories. In the experiments, we investigate the effects of prf on the 20 Newsgroups, Reuters-21578, and Yahoo! Answers datasets using the SVM and κ-NN as classifiers. The results show that the prf scheme outperforms other term weighting schemes, such as term frequency (tf), term frequency and inverse document frequency (tf z.ast; idf), term frequency and relevance frequency (tf z.ast; rf), term frequency and inverse question frequency and question frequency and inverse category frequency (iqf z.ast; qf z.ast; icf). Copyright © 2015 American Scientific Publishers All rights reserved.