Estimating Physical Activity Energy Expenditure from Video

It is well established that health and well-being greatly depends on a person's amount of daily physical activity. Hence accurate measurement of physical activity has been deemed critical in assessing an individual's current health state as well as in predicting future health issues. However, despite its importance, a method for accurately measuring physical activity has remained elusive due to various reasons ranging from technological to individual's adherence and compliance in the measurement task. In this work, a novel automatic method is presented to measure the amount of physical activity directly from video thus bypassing many of the common barriers. Physical activity is measured in terms of the energy expended by a person per minute relative to their resting state using the standard units of metabolic equivalents (METs). The method uses a three-dimensional convolutional neural network trained on videos that captured each subject doing activities of daily living while inside a whole-room calorimeter. The whole-room calorimeter provided gold standard energy expenditure values corresponding to each minute of the video which were used as targets for training the model. The experimental results using leave-one-subject-out cross-validation with seventeen subjects, each with twelve hours of video, showed that the method accurately estimated physical activity energy expenditure with overall root mean squared error of 0.71 METs per minute. This presents promising results to predict physical activity energy expenditure from video and warrants future studies that could be carried out in free-living naturalistic settings.