nnPerf: Demystifying DNN Runtime Inference Latency on Mobile Platforms

We present nnPerf, a real-time on-device profiler designed to collect and analyze the DNN model run-time inference latency on mobile platforms. nnPerf demystifies the hidden layers and metrics used for pursuing I)NN optimizations and adaptations al the granularity of operators and kernels, ensuring every facet contributing to a DNN model's run-time efficiency is easily accessible to mobile developers via well-defined APIs. With nnPerf, the mobile developers can easily identify the bottleneck in model run-time efficiency and optimize the model architecture to meet system -level objectives (SIX)). We implement nnPerf on TFLite framework and evaluate its e2e-, operator-, and kernel -latency profiling accuracy across four mobile platforms. The results show that nnPerf achieves consistently high latency profiling accuracy on both CPU (98.12%) and CPU (99.87%). Our benchmark studies demonstrate that running nnPerf on mobile devices introduces the minlintun overhead to model inference, with 0.231% and 0.605% extra inference latency and power consumption. We further run a case study to show how we leverage nnPerf to migrate OFA, a SOTA NAS system, to kernel oriented model optimization on GPUs.