Comparative analysis of activation functions in neural networks

被引:4
作者
Kamalov, Firuz [1 ]
Nazir, Amril [2 ]
Safaraliev, Murodbek [3 ]
Cherukuri, Aswani Kumar [4 ]
Zgheib, Rita [5 ]
机构
[1] Canadian Univ Dubai, Dept Elect Engn, Dubai, U Arab Emirates
[2] Zayed Univ, Dept Informat Syst, Abu Dhabi, U Arab Emirates
[3] Ural Fed Univ, Automated Elect Syst Dept, Ekaterinburg, Russia
[4] Vellore Inst Technol, Sch IT & Engn, Vellore, Tamil Nadu, India
[5] Canadian Univ Dubai, Dept Comp Sci, Dubai, U Arab Emirates
来源
2021 28TH IEEE INTERNATIONAL CONFERENCE ON ELECTRONICS, CIRCUITS, AND SYSTEMS (IEEE ICECS 2021) | 2021年
关键词
activation function; neural networks; ReLU; sigmoid; loss function;
D O I
10.1109/ICECS53924.2021.9665646
中图分类号
TM [电工技术]; TN [电子技术、通信技术];
学科分类号
0808 ; 0809 ;
摘要
Although the impact of activations on the accuracy of neural networks has been covered in the literature, there is little discussion about the relationship between the activations and the geometry of neural network model. In this paper, we examine the effects of various activation functions on the geometry of the model within the feature space. In particular, we investigate the relationship between the activations in the hidden and output layers, the geometry of the trained neural network model, and the model performance. We present visualizations of the trained neural network models to help researchers better understand and intuit the effects of activation functions on the models.
引用
收藏
页数:6
相关论文
共 16 条
  • [1] Chollet F., 2015, KERAS 20 COMPUTER SO
  • [2] Cybenko G., 1989, Mathematics of Control, Signals, and Systems, V2, P303, DOI 10.1007/BF02551274
  • [3] Multi-type synchronization dynamics of delayed reaction-diffusion recurrent neural networks with discontinuous activations
    Duan, Lian
    Wang, Qian
    Wei, Hui
    Wang, Zengyun
    [J]. NEUROCOMPUTING, 2020, 401 : 182 - 192
  • [4] Frerix Thomas, 2020, Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, P748
  • [5] Kamalov F., 2021, Journal of Computer Science, V17, P251
  • [6] Machine learning based approach to exam cheating detection
    Kamalov, Firuz
    Sulieman, Hana
    Santandreu Calonge, David
    [J]. PLOS ONE, 2021, 16 (08):
  • [7] Forecasting significant stock price changes using neural networks
    Kamalov, Firuz
    [J]. NEURAL COMPUTING & APPLICATIONS, 2020, 32 (23) : 17655 - 17667
  • [8] Understanding convolutional neural networks with a mathematical model
    Kuo, C. -C. Jay
    [J]. JOURNAL OF VISUAL COMMUNICATION AND IMAGE REPRESENTATION, 2016, 41 : 406 - 413
  • [9] Event-triggered stochastic synchronization in finite time for delayed semi-Markovian jump neural networks with discontinuous activations
    Liu, Min
    Wu, Huaiqin
    Zhao, Wei
    [J]. COMPUTATIONAL & APPLIED MATHEMATICS, 2020, 39 (02)
  • [10] Pinkus A., 1999, Acta Numerica, V8, P143, DOI 10.1017/S0962492900002919