Gesture-speech physics in fluent speech and rhythmic upper limb movements

被引:18
作者
Pouw, Wim [1 ,2 ,3 ]
de Jonge-Hoekstra, Lisette [1 ,4 ,5 ]
Harrison, Steven J. [1 ,6 ]
Paxton, Alexandra [1 ,7 ]
Dixon, James A. [1 ,7 ]
机构
[1] Univ Connecticut, Ctr Ecol Study Percept & Act, Storrs, CT USA
[2] Radboud Univ Nijmegen, Donders Inst Brain Cognit & Behav, Heyendaalseweg 135, NL-6525 AJ Nijmegen, Netherlands
[3] Max Planck Nijmegen, Inst Psycholinguist, Nijmegen, Netherlands
[4] Univ Groningen, Fac Behav & Social Sci, Groningen, Netherlands
[5] Royal Dutch Kentalis, St Michielsgestel, Netherlands
[6] Univ Connecticut, Dept Kinesiol, Storrs, CT USA
[7] Univ Connecticut, Dept Psychol Sci, Storrs, CT USA
关键词
hand gesture; speech production; speech acoustics; biomechanics; entrainment; PITCH; COORDINATION; BEHAVIOR; CONTRACTION; PROMINENCE; PERCEPTION; INTONATION; EVOLUTION; PRESSURE; DYNAMICS;
D O I
10.1111/nyas.14532
中图分类号
O [数理科学和化学]; P [天文学、地球科学]; Q [生物科学]; N [自然科学总论];
学科分类号
07 ; 0710 ; 09 ;
摘要
It is commonly understood that hand gesture and speech coordination in humans is culturally and cognitively acquired, rather than having a biological basis. Recently, however, the biomechanical physical coupling of arm movements to speech vocalization has been studied in steady-state vocalization and monosyllabic utterances, where forces produced during gesturing are transferred onto the tensioned body, leading to changes in respiratory-related activity and thereby affecting vocalization F0 and intensity. In the current experiment (n = 37), we extend this previous line of work to show that gesture-speech physics also impacts fluent speech. Compared with nonmovement, participants who are producing fluent self-formulated speech while rhythmically moving their limbs demonstrate heightened F0 and amplitude envelope, and such effects are more pronounced for higher-impulse arm versus lower-impulse wrist movement. We replicate that acoustic peaks arise especially during moments of peak impulse (i.e., the beat) of the movement, namely around deceleration phases of the movement. Finally, higher deceleration rates of higher-mass arm movements were related to higher peaks in acoustics. These results confirm a role for physical impulses of gesture affecting the speech system. We discuss the implications of gesture-speech physics for understanding of the emergence of communicative gesture, both ontogenetically and phylogenetically.
引用
收藏
页码:89 / 105
页数:17
相关论文
共 84 条
[11]   Coordination between breathing and forearm movements during sinusoidal tracking [J].
Ebert, D ;
Rassler, B ;
Hefter, H .
EUROPEAN JOURNAL OF APPLIED PHYSIOLOGY, 2000, 81 (04) :288-296
[12]   Relationship between rhythmic behavior and canonical babbling in infant vocal development [J].
Ejiri, K .
PHONETICA, 1998, 55 (04) :226-237
[13]   Co-occurrence of preverbal vocal behavior and motor action in early infancy [J].
Ejiri, K ;
Masataka, N .
DEVELOPMENTAL SCIENCE, 2001, 4 (01) :40-48
[14]   Prosody in the Auditory and Visual Domains: A Developmental Perspective [J].
Esteve-Gibert, Nuria ;
Guellai, Bahia .
FRONTIERS IN PSYCHOLOGY, 2018, 9
[15]   Prosodic Structure Shapes the Temporal Realization of Intonation and Manual Gesture Movements [J].
Esteve-Gibert, Nuria ;
Prieto, Pilar .
JOURNAL OF SPEECH LANGUAGE AND HEARING RESEARCH, 2013, 56 (03) :850-864
[16]   Understanding the Predictability of Gesture Parameters from Speech and their Perceptual Importance [J].
Ferstl, Ylva ;
Neff, Michael ;
McDonnell, Rachel .
PROCEEDINGS OF THE 20TH ACM INTERNATIONAL CONFERENCE ON INTELLIGENT VIRTUAL AGENTS (ACM IVA 2020), 2020,
[17]  
Feyereisen P., 2017, COGNITIVE PSYCHOL SP
[18]   Multimodal communication and language origins: integrating gestures and vocalizations [J].
Frohlich, Marlen ;
Sievers, Christine ;
Townsend, Simon W. ;
Gruber, Thibaud ;
van Schaik, Carel P. .
BIOLOGICAL REVIEWS, 2019, 94 (05) :1809-1829
[19]   Vocal-tract resonances as indexical cues in rhesus monkeys [J].
Ghazanfar, Asif A. ;
Turesson, Hjalmar K. ;
Maier, Joost X. ;
van Dinther, Ralph ;
Patterson, Roy D. ;
Logothetis, Nikos K. .
CURRENT BIOLOGY, 2007, 17 (05) :425-430
[20]   Learning Individual Styles of Conversational Gesture [J].
Ginosar, Shiry ;
Bar, Amir ;
Kohavi, Gefen ;
Chan, Caroline ;
Owens, Andrew ;
Malik, Jitendra .
2019 IEEE/CVF CONFERENCE ON COMPUTER VISION AND PATTERN RECOGNITION (CVPR 2019), 2019, :3492-3501