BLASER: A Text-Free Speech-to-Speech Translation Evaluation Metric

被引:0
作者
Chen, Mingda [1 ]
Duquenne, Paul-Ambroise [1 ]
Andrews, Pierre [1 ]
Kao, Justine [1 ]
Mourachko, Alexandre [1 ]
Schwenk, Holger [1 ]
Costa-Jussa, Marta R. [1 ]
机构
[1] Meta AI, Menlo Pk, CA 94025 USA
来源
PROCEEDINGS OF THE 61ST ANNUAL MEETING OF THE ASSOCIATION FOR COMPUTATIONAL LINGUISTICS (ACL 2023): LONG PAPERS, VOL 1 | 2023年
关键词
D O I
暂无
中图分类号
TP18 [人工智能理论];
学科分类号
081104 ; 0812 ; 0835 ; 1405 ;
摘要
End-to-End speech-to-speech translation (S2ST) is generally evaluated with text-based metrics. This means that generated speech has to be automatically transcribed, making the evaluation dependent on the availability and quality of automatic speech recognition (ASR) systems. In this paper, we propose a text-free evaluation metric for end-to-end S2ST, named BLASER, to avoid the dependency on ASR systems. BLASER leverages a multilingual multimodal encoder to directly encode the speech segments for source input, translation output and reference into a shared embedding space and computes a score of the translation quality that can be used as a proxy to human evaluation. To evaluate our approach, we construct training and evaluation sets from more than 40k human annotations covering seven language directions. The best results of BLASER are achieved by training with supervision from human rating scores. We show that when evaluated at the sentence level, BLASER correlates significantly better with human judgment compared to ASRdependent metrics including ASR-SENTBLEU in all translation directions and ASR- COMET in five of them. Our analysis shows combining speech and text as inputs to BLASER does not increase the correlation with human scores, but best correlations are achieved when using speech, which motivates the goal of our research. Moreover, we show that using ASR for references is detrimental for text-based metrics.(1)
引用
收藏
页码:9064 / 9079
页数:16
相关论文
共 72 条
[1]  
Anastasopoulos A, 2022, PROCEEDINGS OF THE 19TH INTERNATIONAL CONFERENCE ON SPOKEN LANGUAGE TRANSLATION (IWSLT 2022), P98
[2]  
[Anonymous], 2015, P 10 WORKSHOP STAT M
[3]  
[Anonymous], 2014, P 9 WORKSHOP STAT MA, DOI DOI 10.3115/V1/W14-3348
[4]  
[Anonymous], 2006, P 23 INT C MACH LEAR, DOI DOI 10.1145/1143844.1143891
[5]  
Ardila R, 2020, PROCEEDINGS OF THE 12TH INTERNATIONAL CONFERENCE ON LANGUAGE RESOURCES AND EVALUATION (LREC 2020), P4218
[6]   Massively Multilingual Sentence Embeddings for Zero-Shot Cross-Lingual Transfer and Beyond [J].
Artetxe, Mikel ;
Schwenk, Holger .
TRANSACTIONS OF THE ASSOCIATION FOR COMPUTATIONAL LINGUISTICS, 2019, 7 :597-610
[7]  
Babu A, 2021, Arxiv, DOI arXiv:2111.09296
[8]  
Baevski A., 2020, Advances in neural information processing systems
[9]  
Bapna A., 2022, arXiv
[10]  
Besacier L, 2022, Arxiv, DOI arXiv:2210.11835