Novel multi-convolutional neural network fusion approach for smile recognition

The smile is one of the most common human facial expressions encountered in our daily lives. Smile recognition can be used in many scenarios, such as emotion monitoring, human-to-robot games, and camera shutter control, which is why smile recognition has received significant attention of researchers. This topic is a significant but challenging problem, particularly in unconstrained scenarios. The variety of facial sizes, illumination conditions, head poses, occlusions, and other factors increases the difficulty of this problem. To address this problem, we propose a novel multiple convolutional neural network (CNN) fusion approach in which a face-based CNN and a mouth-based CNN are used to perform smile recognition. According to the results obtained using the two CNNs, we fuse the two networks using a specified weight and choose the higher-probability result as the final result. Experimental results indicate that the method is effective on a real-world smile dataset (GENKI-4K). The smile recognition rate of the proposed method is improved by 1.6% and 3.3% relative to the face-based CNN and mouth-based CNN, respectively, and the proposed method outperforms the most of previous methods.