An FPGA-Based Microinstruction Sequence Driven Spaceborne Convolution Neural Network Accelerator

Recently, with the evolvement of space remote sensing technology, the main earth observation device has been gradually transitioning from the single-satellite to a constellation composed of light and small satellites. A constellation of several high-resolution satellites collects hundreds of TBs(Terabytes) of RSI(Remote Sensing Image) data every day. The traditional satellite-to-ground data transmission mechanism has been unable to match the massive remote sensing data processing. In-orbit satellites need to improve their data processing capabilities to deal with increasingly complex observation missions. Meanwhile, in the field of RSI processing, deep learning algorithms based on CNN(Convolutional Neural Network) have become the mainstream method due to their excellent performance. However, the computation-intensive and memory-intensive features have brought many challenges to the deployment of CNN. Academia and industry propose many specific acceleration methods for the CNN domain to cope with the various application scenarios. Numerous FPGA(Field Programmable Gate Array) and ASIC(Application Specific Integrated Circuit) accelerators have been designed to accelerate CNN in edge and data center scenarios. Compared with ASIC, FPGA has higher flexibility and faster development iteration speed, making it very suitable for spaceborne scenarios. In this paper, we propose a microinstruction driven CNN Accelerator for RSI processing on FPGA. This accelerator is jointly designed by software and hardware, which mainly optimizes microinstruction coding, instruction-level parallelism(Coarse-Grained Parallelism) and operation-level parallelism(Fine-Grained Parallelism) under the constraints of limited storage bandwidth and computing resources on satellites. At software level, we propose an extensible microinstruction encoding format and the corresponding compilation method(Micro Assembler). A microinstruction code covers 14 instructions in 4 types, which can schedule the dataflow between different components of the accelerator. The micro assembler performs graph-level optimization on the CNN topology by convolutional loop tiling and operator fusion, and then generates micro-instruction sequences that can be executed by the accelerator. At hardware level, we design and implement an RTL(Register Transfer Level) CNN accelerator, which is mainly composed of micro controller and logic operator. The micro controller achieves the parallel execution of different types of instruction by a 5-stage coarse-grained pipeline(Data Load, Data Fetch, Compute, Post Process, Write Back). The logic operator is a computing array with DSP48E1 hard core resources cascaded, which can achieve parallel execution of convolution operations by a 32-stage fine-grained pipeline. When the pipeline is established, the logic operator can complete 32×32 MAC(Multiply-accumulate) operations in one clock cycle. The performance of our proposed accelerator is evaluated on the Xilinx VX690T FPGA chip commonly found on satellites. The designed power consumption is 10.68W. The RMT(Runtime Max Throughput) reaches 378.63GOP/s, and the ME(MAC Efficiency) reaches 91.5%. When our accelerator is used as a coprocessor to accelerate the CNN object detection algorithm YOLOV3Tiny, the average accuracy of the RSI data set reaches 0.9 and the detection speed reaches 102frames/s. The evaluation results show that our accelerator is 14 times more energy efficiency than the typical GPU acceleration method, and has more than 6.9% improvement in ME compared with other FPGA accelerators. © 2022, Science Press. All right reserved.