Multimodal emotion classification using machine learning in immersive and non-immersive virtual reality

被引:2
作者
Lima, Rodrigo [1 ,2 ,3 ]
Chirico, Alice [4 ]
Varandas, Rui [6 ,7 ]
Gamboa, Hugo [6 ,7 ]
Gaggioli, Andrea [4 ,5 ]
Badia, Sergi Bermudez [1 ,2 ,3 ]
机构
[1] Univ Madeira, Fac Ciencias Exatas & Engn, Campus Univ Penteada, P-9000105 Funchal, Portugal
[2] Univ Nova Lisboa, Fac Ciencias & Tecnol, NOVA Lab Comp Sci & Informat, P-2829516 Caparica, Portugal
[3] Agencia Reg Desenvolvimento Invest Tecnol & Inovac, Caminho Penteada, P-9020105 Funchal, Portugal
[4] Univ Cattolica Sacro Cuore, Dipartimento Psicol, Largo Agostino Gemelli 1, I-20123 Milan, Italy
[5] Ist Auxol Italiano, IRCCS, Appl Technol Neuropsychol Lab, I-20149 Milan, Italy
[6] Univ Nova Lisboa, Fac Ciencias & Tecnol, Lab Instrumentat Biomed Engn & Radiat Phys, LIBPhys, P-2829516 Setubal, Portugal
[7] PLUX Wireless Biosignals SA, Ave 5 Outubro 70, P-1050059 Lisbon, Portugal
关键词
Affective computing; Emotions; Wearables; Physiological signals; Machine learning; Virtual reality; RECOGNITION; APPRAISAL; DATABASE; VALENCE; STRESS;
D O I
10.1007/s10055-024-00989-y
中图分类号
TP39 [计算机的应用];
学科分类号
081203 ; 0835 ;
摘要
Affective computing has been widely used to detect and recognize emotional states. The main goal of this study was to detect emotional states using machine learning algorithms automatically. The experimental procedure involved eliciting emotional states using film clips in an immersive and non-immersive virtual reality setup. The participants' physiological signals were recorded and analyzed to train machine learning models to recognize users' emotional states. Furthermore, two subjective ratings emotional scales were provided to rate each emotional film clip. Results showed no significant differences between presenting the stimuli in the two degrees of immersion. Regarding emotion classification, it emerged that for both physiological signals and subjective ratings, user-dependent models have a better performance when compared to user-independent models. We obtained an average accuracy of 69.29 +/- 11.41% and 71.00 +/- 7.95% for the subjective ratings and physiological signals, respectively. On the other hand, using user-independent models, the accuracy we obtained was 54.0 +/- 17.2% and 24.9 +/- 4.0%, respectively. We interpreted these data as the result of high inter-subject variability among participants, suggesting the need for user-dependent classification models. In future works, we intend to develop new classification algorithms and transfer them to real-time implementation. This will make it possible to adapt to a virtual reality environment in real-time, according to the user's emotional state.
引用
收藏
页数:23
相关论文
共 50 条
[31]   Effects of physical, non-immersive virtual, and immersive virtual store environments on consumers' perceptions and purchase behavior [J].
Lombart, Cindy ;
Millan, Elena ;
Normand, Jean-Marie ;
Verhulst, Adrien ;
Labbe-Pinlon, Blandine ;
Moreau, Guillaume .
COMPUTERS IN HUMAN BEHAVIOR, 2020, 110
[32]   An intradialytic non-immersive virtual reality exercise programme: a crossover randomized controlled trial [J].
Martinez-Olmos, Francisco J. ;
Gomez-Conesa, Antonia A. ;
Garcia-Testal, Alicia ;
Ortega-Perez-de-Villar, Lucia ;
Valtuena-Gimeno, Noemi ;
Gil-Gomez, Jose A. ;
Garcia-Maset, Rafael ;
Segura-Orti, Eva .
NEPHROLOGY DIALYSIS TRANSPLANTATION, 2022, 37 (07) :1366-1374
[33]   Learning Science in Immersive Virtual Reality [J].
Parong, Jocelyn ;
Mayer, Richard E. .
JOURNAL OF EDUCATIONAL PSYCHOLOGY, 2018, 110 (06) :785-797
[34]   Effectiveness of Non-Immersive Virtual Reality Simulation in Learning Knowledge and Skills for Nursing Students: Meta-analysis [J].
Qiao, Jia ;
Huang, Can-Ran ;
Liu, Qian ;
Li, Su-Ya ;
Xu, Jing ;
Li, Lu ;
Redding, Sharon R. ;
Ouyang, Yan-Qiong .
CLINICAL SIMULATION IN NURSING, 2023, 76 :26-38
[35]   Short-term motor learning through non-immersive virtual reality task in individuals with down syndrome [J].
Carlos Bandeira de Mello Monteiro ;
Talita Dias da Silva ;
Luiz Carlos de Abreu ;
Felipe Fregni ;
Luciano Vieira de Araujo ;
Fernando Henrique Inocêncio Borba Ferreira ;
Claudio Leone .
BMC Neurology, 17
[36]   Advancing emotion recognition with Virtual Reality: A multimodal approach using physiological signals and machine learning [J].
Polo, Edoardo Maria ;
Iacomi, Francesco ;
Rey, Alberto Valdes ;
Ferraris, Davide ;
Paglialonga, Alessia ;
Barbieri, Riccardo .
Computers in Biology and Medicine, 2025, 193
[37]   SeamlessVR: Bridging the Immersive to Non-Immersive Visualization Divide [J].
Liao, Shuqi ;
Chaudhri, Sparsh ;
Karwa, Maanas K. ;
Popescu, Voicu .
IEEE TRANSACTIONS ON VISUALIZATION AND COMPUTER GRAPHICS, 2025, 31 (05) :2806-2816
[38]   Enhancing the Potential of Machine Learning for Immersive Emotion Recognition in Virtual Environment [J].
Abinaya, M. ;
Vadivu, G. .
EAI ENDORSED TRANSACTIONS ON SCALABLE INFORMATION SYSTEMS, 2024, 11 (04) :1-9
[39]   Identifying subassemblies and understanding their functions during a design review in immersive and non-immersive virtual environments [J].
Fanika Lukačević ;
Stanko Škec ;
Peter Törlind ;
Mario Štorga .
Frontiers of Engineering Management, 2021, 8 :412-428
[40]   Prototyping and Validating a Non-immersive Virtual Reality Serious Game for Healthcare Fire Safety Training [J].
Rahouti, Anass ;
Lovreglio, Ruggiero ;
Datoussaid, Selim ;
Descamps, Thierry .
FIRE TECHNOLOGY, 2021, 57 (06) :3041-3078