Learnable Leaky ReLU (LeLeLU): An Alternative Accuracy-Optimized Activation Function

被引：44

作者：

Maniatopoulos, Andreas ^{[1
]}

Mitianoudis, Nikolaos ^{[1
]}

机构：

[1] Democritus Univ Greece, Dept Elect & Comp Engn, Xanthi 67100, Greece

来源：

INFORMATION | 2021年 / 12卷 / 12期

关键词：

activation function; ReLU family; activation function test;

D O I：

10.3390/info12120513

中图分类号：

TP [自动化技术、计算机技术];

学科分类号：

0812 ;

摘要：

In neural networks, a vital component in the learning and inference process is the activation function. There are many different approaches, but only nonlinear activation functions allow such networks to compute non-trivial problems by using only a small number of nodes, and such activation functions are called nonlinearities. With the emergence of deep learning, the need for competent activation functions that can enable or expedite learning in deeper layers has emerged. In this paper, we propose a novel activation function, combining many features of successful activation functions, achieving 2.53% higher accuracy than the industry standard ReLU in a variety of test cases.

引用

页数：16