SpikeMotion: A Transformer Framework for High-Throughput Video Segmentation on FPGA

Transformers have emerged at the forefront in the training and inference of diverse machine learning tasks, encompassing video processing, image generation and classification, and natural language processing (NLP). Despite their increasing prevalence, a comprehensive framework to efficiently implement them has been lacking. This study introduces a transformer-based framework which accelerates image processing of UNETR (U-shaped neural network transformer) model for video segmentation task using the cityscapes dataset. Given the large size of images in the dataset we incorporate hyperattention and mixed precision in our design. Our model is trained on Google A1OO GPU accelerator and profiled. Finally, our design is implemented on FPGA to take advantage of the reconfigurable and high-throughput characteristics of system-on-chips (SoC) for image processing. Our results indicate improvements compared to existing research in this domain.