Detecting Deception from Gaze and Speech Using a Multimodal Attention LSTM-Based Framework

被引:17
作者
Gallardo-Antolin, Ascension [1 ]
Montero, Juan M. [2 ]
机构
[1] Univ Carlos III Madrid, Dept Signal Theory & Commun, Avda Univ 30, Madrid 28911, Spain
[2] Univ Politecn Madrid, ETSIT, Speech Technol Grp, Avda Complutense 30, Madrid 28040, Spain
来源
APPLIED SCIENCES-BASEL | 2021年 / 11卷 / 14期
关键词
deception detection; multimodal; gaze; speech; LSTM; attention; fusion; SYSTEM;
D O I
10.3390/app11146393
中图分类号
O6 [化学];
学科分类号
0703 ;
摘要
The automatic detection of deceptive behaviors has recently attracted the attention of the research community due to the variety of areas where it can play a crucial role, such as security or criminology. This work is focused on the development of an automatic deception detection system based on gaze and speech features. The first contribution of our research on this topic is the use of attention Long Short-Term Memory (LSTM) networks for single-modal systems with frame-level features as input. In the second contribution, we propose a multimodal system that combines the gaze and speech modalities into the LSTM architecture using two different combination strategies: Late Fusion and Attention-Pooling Fusion. The proposed models are evaluated over the Bag-of-Lies dataset, a multimodal database recorded in real conditions. On the one hand, results show that attentional LSTM networks are able to adequately model the gaze and speech feature sequences, outperforming a reference Support Vector Machine (SVM)-based system with compact features. On the other hand, both combination strategies produce better results than the single-modal systems and the multimodal reference system, suggesting that gaze and speech modalities carry complementary information for the task of deception detection that can be effectively exploited by using LSTMs.
引用
收藏
页数:16
相关论文
共 44 条
[1]  
Abadi M., 2015, P 12 USENIX S OPERAT
[2]   Detecting Deceptive Behavior via Integration of Discriminative Features From Multiple Modalities [J].
Abouelenien, Mohamed ;
Perez-Rosas, Veronica ;
Mihalcea, Rada ;
Burzo, Mihai .
IEEE TRANSACTIONS ON INFORMATION FORENSICS AND SECURITY, 2017, 12 (05) :1042-1055
[3]  
[Anonymous], 2010, OPEN GAZE API GAZEPO
[4]  
[Anonymous], 2015, P 2015 C EMP METH NA
[5]   MultiModal Deception Detection: Accuracy, Applicability and Generalizability [J].
Belavadi, Vibha ;
Zhou, Yan ;
Bakdash, Jonathan Z. ;
Kantarcioglu, Murat ;
Krawczyk, Daniel C. ;
Nguyen, Linda ;
Rakic, Jelena ;
Thuriasingham, Bhavani .
2020 SECOND IEEE INTERNATIONAL CONFERENCE ON TRUST, PRIVACY AND SECURITY IN INTELLIGENT SYSTEMS AND APPLICATIONS (TPS-ISA 2020), 2020, :99-106
[6]  
Benus S., 2006, Proceedings of ISCA 3rd International Conference on Speech Prosody, DOI 10.7916/D8SQ97TG
[7]   In the Eye of the Deceiver: Analyzing Eye Movements as a Cue to Deception [J].
Borza, Diana ;
Itu, Razvan ;
Danescu, Radu .
JOURNAL OF IMAGING, 2018, 4 (10)
[8]  
Chollet F., 2015, Keras
[9]  
Chorowski J, 2015, ADV NEUR IN, V28
[10]   Cues to deception [J].
DePaulo, BM ;
Lindsay, JJ ;
Malone, BE ;
Muhlenbruck, L ;
Charlton, K ;
Cooper, H .
PSYCHOLOGICAL BULLETIN, 2003, 129 (01) :74-118