Towards Efficient Inference on Mobile Device via Pruning

Mobile edge intelligence requires deploying deep learning neural network models on mobile devices. Model sparsity techniques, including low-precision quantization and pruning, have been proposed to achieve higher model compression rates at a lower cost of model accuracy degradation. However, these sparsity techniques still face challenges in deployment on mobile devices. Due to the lack of capability for fine-tuning model weight parameters in mobile inference frameworks, any pruning technique must be carefully applied. The deployment of sparsity techniques in mobile inference frameworks is challenging. To address these challenges, we propose PoI (Pruning on Inference), the inference framework, which integrates lowprecision quantization and pruning optimization for inference computation, enabling efficient deployment of deep learning models on mobile devices. The design of this framework is based on the observation that memory access patterns have a crucial impact on inference latency during model execution. This provides us with an opportunity to perform efficient pruning operations during inference execution. To realize this idea, we have designed an inference computation framework. Firstly, it supports different quantization bit-widths to fully exploit the computational capabilities of different hardware devices. After the user specifies the inference model, it checks the current device's support for various quantization bit-widths and sets the corresponding compilation optimization options. During the actual execution process, it can select an appropriate pruning ratio based on the user-specified tolerance for accuracy loss. In the specific operator computation execution process, it performs pruning operations on the corresponding ratio of intra-kernel computations. At the same time, this pruning takes into account the potential impact on memory access patterns to minimize the additional overhead caused by cache misses. We evaluated our implementation based on commonly used computer vision neural network models and popular mobile devices in the real world. Compared to current mainstream solutions, our implementation can reduce the average inference latency of models. Moreover, our implementation can be easily migrated to other popular mobile devices.