DIR-BHRNet: A Lightweight Network for Real-Time Vision-Based Multiperson Pose Estimation on Smartphones

Human pose estimation (HPE), particularly multiperson pose estimation (MPPE), has been applied in many domains, such as human-machine systems. However, the current MPPE methods generally run on powerful GPU systems and take a lot of computational costs. Real-time MPPE on mobile devices with low-performance computing is a challenging task. In this article, we propose a lightweight neural network, DIR-BHRNet, for real-time MPPE on smartphones. In DIR-BHRNet, we design a novel lightweight convolutional module, dense inverted residual (DIR), to improve accuracy by adding a depthwise convolution and a shortcut connection into the well-known inverted residual, and a novel efficient neural network structure, balanced HRNet (BHRNet), to reduce computational costs by reconfiguring the proper number of convolutional blocks on each branch. We evaluate DIR-BHRNet on the well-known COCO and CrowdPose datasets. The results show that DIR-BHRNet outperforms the state-of-the-art methods in terms of accuracy with a real-time computational cost. Finally, we implement the DIR-BHRNet on the current mainstream Android smartphones, which perform more than 10 FPS. The free-used executable file (Android 10), source code, and a video description of this work are publicly available on the page(1) to facilitate the development of real-time MPPE on smartphones.