ATA: Attentional Non-Linear Activation Function Approximation for VLSI-Based Neural Networks

In this letter, we present an attentional non-linear activation function approximation method called ATA for VLSI-based neural networks. Unlike other approximation methods that pursue the low hardware resources with a high recognition accuracy loss, the ATA utilizes the pixel attention to focus on the important features to keep the recognition accuracy and reduce resource cost. Specifically, attention applied in the activation function is realized by the approximated activation functions with different fitting errors for VLSI-based neural networks. The important features are highlighted by the piecewise linear function and improved look-up table with low fitting error, while the trivial features are ignored with the large fitting error. Experimental results demonstrate that the ATA outperforms other state-of-the-art approximation methods in recognition accuracy, power and area.