DIF-LUT: A Simple Yet Scalable Approximation for Non-linear Activation Function on FPGA

Non-linear activation function plays an essential role in neural networks (NNs) for their generalization ability. However, deploying the intricate mathematical operations on hardware platforms like Field-Programmable Gate Array (FPGA) turns out a great challenge. Prior works based on piecewise functions or look-up table (LUT) either involve complex manual operations or neglect hardware overhead. To this end, this paper proposes a simple yet scalable and effective approximation called DIF-LUT, which is applicable to various non-linear functions. Specifically, the proposed method can achieve accurate approximation by piecewise linear matching to fit the function derivative roughly and range addressable LUT to offset the difference. Moreover, self-adaptive mechanisms are applied to automatically minimize hardware cost in terms of different accuracies. The experiments show that compared to state-of-the-art methods, DIF-LUT costs 43.68% fewer LUTs and 70.8% fewer flip-flops (FFs) without any digital signal processor (DSP), while achieving 2.7x approximation accuracy at 554.1MHz on Xilinx Zynq UltraScale+.