OFMT-net: Advancing micro-expression recognition through optical flow and multi-task network with self-attention

Micro-expression is difficult to recognize due to short duration and subtle action range, but it contains rich and real psychological information, which has important research value in criminal investigation, teaching and other fields. In response to issues like limited facial expression dynamics, suboptimal feature extraction, and susceptibility to overfitting, we proposed a micro-expression recognition method based on optical flow and multi-task convolutional neural network (OFMT-Net). It capitalizes on optical flow data from onset to apex frames as input. Feature extraction is conducted through a shared-parameter network, funnelling outputs into a dual-tower network designed for emotional and Action Unit (AU) recognition. This network incorporates a self-attention mechanism for effective classification, driven by a dual weighted loss function. The method fully extracts the relevant information contained in the facial action unit, and uses the implicit data enhancement advantages of the multi-task framework to improve the recognition accuracy and reduce the sample dependence problems. Cross-validation results on the joint dataset demonstrate that the model achieves an accuracy rate of 79.89%, an unweighted average recall rate of 75.05%, and an unweighted F1 score of 75.08%, surpassing many mainstream models. The related code is publicly available at https://github.com/WenyuanLi001/OFMT-Net