A Preliminary Study on Class Probability Estimation for Random Forest Using Kernel Density estimators

Random forest cannot give accurate and calibrated posterior class probability estimates for its predictions. In this paper, we propose novel probabilities estimators combining random forests with kernel density estimation. Kernel density estimator can manage to obtain smooth non-parametric estimations of class probabilities, but fail to scale up to the high dimensional data. In order to apply kernel density estimator to high dimensional data, we proposed to utilize random forest for dimension reduction. First, a random forest model is built with the training data. Secondly, for the test instance, we perform local kernel density estimators in the reduced subspaces corresponding to trees of random forest, and then average the estimated probabilities over the trees. Preliminary experiments on synthetic high-dimensional data showed that the new method provided accurate probability estimates for the output of the random forest.