Sparse Least Squares Support Vector Regression via Multiresponse Sparse Regression

Least square support vector machines (LSSVMs) are an alternative to SVMs because the training process for LSSVMs is based on solving a linear equation system while the training process for SVMs relies on solving a quadratic programming optimization problem. When LSSVMs are dealing with regression tasks, we refer to them as Least square support vector regressors (LSSVRs). Despite solving a linear system is easier than solving a quadratic programming optimization problem, the absence of sparsity in the Lagrange multiplier vector obtained after training a LSSVR model is an important drawback. To overcome this drawback, we present a new approach for sparse LSSVR called Optimally Pruned LSSVR (OP-LSSVR). Our proposal relies on a ranking method, named Multiresponse Sparse Regression (MRSR), which is used to sort the patterns in terms of relevance. After doing so, the leave-one-out (LOO) criterion is also used in order to select an appropriate number of support vectors. Our proposal was inspired by a recent methodology called OP-ELM, which prunes neurons in the hidden layer of Extreme Learning Machines. Therefore, in this paper, we put LSSVR and MRSR to work together in order to achieve sparse regressors while we achieved equivalent (or even superior) performance for real-world regression tasks.