RT3D: Achieving Real-Time Execution of 3D Convolutional Neural Networks on Mobile Devices

被引:0
|
作者
Niu, Wei [1 ]
Sun, Mengshu [2 ]
Li, Zhengang [2 ]
Chen, Jou-An [3 ]
Guan, Jiexiong [1 ]
Shen, Xipeng [3 ]
Wang, Yanzhi [2 ]
Liu, Sijia [4 ]
Lin, Xue [2 ]
Ren, Bin [1 ]
机构
[1] William & Mary, Williamsburg, VA 23185 USA
[2] Northeastern Univ, Boston, MA 02115 USA
[3] North Carolina State Univ, Raleigh, NC 27695 USA
[4] Michigan State Univ, E Lansing, MI 48824 USA
来源
THIRTY-FIFTH AAAI CONFERENCE ON ARTIFICIAL INTELLIGENCE, THIRTY-THIRD CONFERENCE ON INNOVATIVE APPLICATIONS OF ARTIFICIAL INTELLIGENCE AND THE ELEVENTH SYMPOSIUM ON EDUCATIONAL ADVANCES IN ARTIFICIAL INTELLIGENCE | 2021年 / 35卷
基金
美国国家科学基金会;
关键词
D O I
暂无
中图分类号
TP18 [人工智能理论];
学科分类号
081104 ; 0812 ; 0835 ; 1405 ;
摘要
Mobile devices are becoming an important carrier for deep learning tasks, as they are being equipped with powerful, high-end mobile CPUs and GPUs. However, it is still a challenging task to execute 3D Convolutional Neural Networks (CNNs) targeting for real-time performance, besides high inference accuracy. The reason is more complex model structure and higher model dimensionality overwhelm the available computation/storage resources on mobile devices. A natural way may be turning to deep learning weight pruning techniques. However, the direct generalization of existing 2D CNN weight pruning methods to 3D CNNs is not ideal for fully exploiting mobile parallelism while achieving high inference accuracy. This paper proposes RT3D, a model compression and mobile acceleration framework for 3D CNNs, seamlessly integrating neural network weight pruning and compiler code generation techniques. We propose and investigate two structured sparsity schemes i.e., the vanilla structured sparsity and kernel group structured (KGS) sparsity that are mobile acceleration friendly. The vanilla sparsity removes whole kernel groups, while KGS sparsity is a more fine-grained structured sparsity that enjoys higher flexibility while exploiting full on-device parallelism. We propose a reweighted regularization pruning algorithm to achieve the proposed sparsity schemes. The inference time speedup due to sparsity is approaching the pruning rate of the whole model FLOPs (floating point operations). RT3D demonstrates up to 29.1 x speedup in end-to-end inference time comparing with current mobile frameworks supporting 3D CNNs, with moderate 1% similar to 1.5% accuracy loss. The endto-end inference time for 16 video frames could be within 150 ms, when executing representative C3D and R(2+1)D models on a cellphone. For the first time, real-time execution of 3D CNNs is achieved on off-the-shelf mobiles.
引用
收藏
页码:9179 / 9187
页数:9
相关论文
共 50 条
  • [41] Resource Efficient 3D Convolutional Neural Networks
    Koepueklue, Okan
    Kose, Neslihan
    Gunduz, Ahmet
    Rigoll, Gerhard
    2019 IEEE/CVF INTERNATIONAL CONFERENCE ON COMPUTER VISION WORKSHOPS (ICCVW), 2019, : 1910 - 1919
  • [42] LipsID Using 3D Convolutional Neural Networks
    Hlavac, Miroslav
    Gruber, Ivan
    Zelezny, Milos
    Karpov, Alexey
    SPEECH AND COMPUTER (SPECOM 2018), 2018, 11096 : 209 - 214
  • [43] A real-time 3D multi-view rendering from a real-time 3D capture
    1600, Blackwell Publishing Ltd (44):
  • [44] Real-Time 3D Mapping Aided GNSS on Android Devices
    Adjrad, Mounir
    Groves, Paul D.
    PROCEEDINGS OF THE 31ST INTERNATIONAL TECHNICAL MEETING OF THE SATELLITE DIVISION OF THE INSTITUTE OF NAVIGATION (ION GNSS+ 2018), 2018, : 345 - 356
  • [45] A Real-time 3D Surround View Pipeline for Embedded Devices
    Eker, Onur
    Ercan, Burak
    Bayraktar, Berkant
    Bal, Murat
    PROCEEDINGS OF THE 17TH INTERNATIONAL JOINT CONFERENCE ON COMPUTER VISION, IMAGING AND COMPUTER GRAPHICS THEORY AND APPLICATIONS (VISAPP), VOL 4, 2022, : 257 - 263
  • [46] Optimizing on Real-time Fluid 3D Effect in Mobile Environment
    Lu, Jue-min
    Piao, Jin-chun
    Kim, Shin-dug
    ACM IMCOM 2015, Proceedings, 2015,
  • [47] T-C3D: Temporal Convolutional 3D Network for Real-Time Action Recognition
    Liu, Kun
    Liu, Wu
    Gan, Chuang
    Tan, Mingkui
    Ma, Huadong
    THIRTY-SECOND AAAI CONFERENCE ON ARTIFICIAL INTELLIGENCE / THIRTIETH INNOVATIVE APPLICATIONS OF ARTIFICIAL INTELLIGENCE CONFERENCE / EIGHTH AAAI SYMPOSIUM ON EDUCATIONAL ADVANCES IN ARTIFICIAL INTELLIGENCE, 2018, : 7138 - 7145
  • [48] Real-Time 3D Pointing Gesture Recognition in Mobile Space
    Park, Chang-Beom
    Roh, Myung-Cheol
    Lee, Seong-Whan
    2008 8TH IEEE INTERNATIONAL CONFERENCE ON AUTOMATIC FACE & GESTURE RECOGNITION (FG 2008), VOLS 1 AND 2, 2008, : 520 - 525
  • [49] Text3D: 3D Convolutional Neural Networks for Text Classification
    Wang, Jinrui
    Li, Jie
    Zhang, Yirui
    ELECTRONICS, 2023, 12 (14)
  • [50] Real-Time Gesture Recognition Using 3D Sensory Data and a Light Convolutional Neural Network
    Diliberti, Nicholas
    Peng, Chao
    Kauffman, Christopher
    Dong, Yangzi
    Hansberger, Jeffrey T.
    PROCEEDINGS OF THE 27TH ACM INTERNATIONAL CONFERENCE ON MULTIMEDIA (MM'19), 2019, : 401 - 410