Profiling Power Consumption for Deep Learning on Resource Limited Devices

The introduction of convolutional neural networks (CNN) has had a significant impact on various computer vision tasks. The process of inference, where a CNN takes images as input and produces corresponding predictions, is a complex and resource hungry task that consumes significant power. Originally much of the processing was done on well resourced machines locally or hosted on various cloud platforms but there has been a recent trend towards moving the processing of data closer to where it is produced within resource limited 'edge' devices. It is important to understand the implications and limitations for deployment of a CNN to a device such as an earth observation satellite, which does not have a constant power source. Quantisation is a model optimising technique, where the precision of weights, biases, and activations are reduced, such that they consume less memory and power, and is a common approach to facilitate such deployments. This paper investigates the power consumption behaviour of CNN models from the DenseNet, EfficientNet, MobileNet, ResNet, ConvNeXt & RegNet architecture families, processing imagery on board a Nvidia Jetson Orin Nano platform. It was found that energy consumption varied from 6 mJ to 26 mJ per image for different base (non-quantised) models. Accuracy varied from 69% to 82% and latency varied from 1.3 ms to 7.5 ms per image. The effectiveness of quantisation in reducing the power requirements of CNNs during inference was also investigated, focusing on the use case of deployment to an earth observation satellite. A large difference was found between architectures with some reducing the energy consumption by up to 87% while others achieve less than 10%. The metrics "accuracy-per-joule" and "latency-by-joule" were introduced and used to benchmark and more effectively compare models energy-effectiveness and the impact of quantisation. After quantisation, an improvement in accuracy-per-joule of up to 700% and a latency-by-joule reduction of 99% was achieved.