Transformer-OPU: An FPGA-based Overlay Processor for Transformer Networks

Existing implementations of transformer networks by field-programmable gate array (FPGA) focus only on attention computation, or suffer from fixed model structure without flexibility. In this article, we propose an FPGA-based overlay processor, named Transformer-OPU for general accelerations of transformer networks. Experimental result shows that our Transformer-OPU achieves 5.19-15.06x and 1.14-2.89x speedup compared with CPU and GPU, respectively. We also observe 1.10-2.47x better latency compared with previously customized FPGA accelerators, and is 1.45x faster than NPE.