Fast Activation Function Approach for Deep Learning Based Online Anomaly Intrusion Detection

The piecewise-linear activation functions such as ReLU become the catalyst that revolutionizes the training of the deep neural networks. Common nonlinear activation functions used in neural networks such as the tanh and the sigmoid activation functions suffer from saturation during training. The saturation behavior causes the problem of vanishing stochastic gradient decent. We propose a fast activation function, namely the Adaptive Linear Function (ALF) to increase the convergence speed and accuracy of the deep leaning structure for real-time applications. The ALF reduces the saturation effects caused by the soft activation functions and the vanishing gradient caused by the negative values of the ReLU. We evaluate the training method for an online anomaly intrusion detection system using Deep Belief Network (DBN) and simulating four benchmark datasets. The activation function increases the convergence speed of the DBN, with the entire training time reduced 80% compared to the sigmoid, ReLU, and tanh activation functions. The method achieves an accuracy rate of 98.59% on the total 10% KDDCUP'99 test dataset, 96.2% on the NSL-KDD dataset, 98.4% on the Kyoto dataset, and 96.57% on the CSIC HTTP dataset. The proposed activation function outperformed the results obtained when any of the three activation functions-sigmoid, ReLu, or tanh-was used on the test stream of the four datasets. Furthermore, the DBN structure outperforms state-of-the-art networks such as the Stacked Sparse AutoEncoder Based Extreme Learning Machine (SSAELM) in both accuracy and convergence speed.