Remote heart rate estimation via convolutional neural networks with transformers

Remote photoplethysmography (rPPG) uses subtle color changes in facial videos to estimate heart rate(HR). However, recent methods face challenges in solving remote heart rate estimation tasks, because the color changes in facial skin are very subtle and the pseudo-periodicity of rPPG requires long-distance temporal detection. To address these issues of the rPPG estimation task, we propose a convolution neural network with transformers for rPPG estimation which takes the advantages of convolutions in the locality and transformers in long-range dependencies. Specifically, we first proposed a local feed-forward module following the multi-head self-attention to compensate for the difficulties of transformers in capturing neighboring feature information. Then we add the relative and absolute position encoding to obtain the ordering of the tokens which is the key to capturing the pseudo-periodicity of rPPG. Furthermore, the temporal multi-scale module is proposed to learn the temporal information from different scales. Extensive experimental results demonstrate that our method is superior to the state-of-the-art results on the COHFACE, UBFC-rPPG, PURE, and VIPL-HR databases compared with traditional methods and deep learning-based methods. We also evaluated our method under different conditions on the VIPL-HR database, and the results show that our method is robust to various conditions. (c) 2023 The Franklin Institute. Published by Elsevier Inc. All rights reserved.