Hardware Acceleration of Capsule Networks for Real-time Applications

Capsule networks (CapsNet) are a category of deep learning neural networks (DNN) that address one of the main issues and deficiencies of convolutional neural networks (CNN); loss of spatial information in pooling layers. However, the main concern with CapsNets is their compute-intensive nature, which is mainly related to the vector-based calculations during dynamic routing and acts as a barrier for their deployment in real-time applications. To address the specific computing requirements of dynamic routing in capsule layers of CapsNets, we develop a hardware accelerator for dynamic routing using Vitis HLS. In this paper, we present a hardware acceleration solution for capsule networks by integrating AMD Xilinx deep processing unit (DPU) and a custom accelerator for the capsule layer. Our results show significant improvement in throughput compared with the baseline implementation when CapsNet is implemented on Zynq UltraScale+ MPSoC ZCU102 using Vitis AI DPUs.