Improvement of robustness against electrode shift for facial electromyogram-based facial expression recognition using domain adaptation in VR-based metaverse applications

Recognition of users' facial expressions and reflecting them on the face of the user's virtual avatar is a key technology for realizing immersive virtual reality (VR)-based metaverse applications. As a method to realize this technology, a facial electromyogram (fEMG)-based facial expression recognition (FER) system, with the fEMG electrodes being attached on the pad of a VR headset, has recently been proposed. However, the performance of such FER systems has severely deteriorated when the locations of fEMG electrodes change by the re-wearing of the VR headset, requiring long and tedious calibration sessions every time the user wears the VR headset. In this study, we developed an fEMG-based FER system that is robust against electrode shifts by employing new signal processing techniques: covariate shift adaptation techniques in feature and classifier domains. To verify the feasibility of the proposed method, fEMG data were recorded while participants were making 11 facial expressions repeatedly in four sessions, between which they detached and reattached the fEMG electrodes on their faces. Our experiments showed that classification accuracy dropped from 88 to 79% by the change of the electrode locations when the proposed method was not applied, whereas the accuracy was significantly improved up to 86% when the proposed covariate shift adaptation method was employed. It is expected that the proposed method would contribute to enhancing the practicality of the fEMG-based FER, promoting the practical application of the fEMG-based FER to VR-based metaverse applications.