Transformer-based Conformal Predictors for Paraphrase Detection

被引:0
作者
Giovannotti, Patrizio [1 ]
Gammerman, Alex [1 ]
机构
[1] Royal Holloway Univ London, Egham, Surrey, England
来源
CONFORMAL AND PROBABILISTIC PREDICTION AND APPLICATIONS, VOL 152 | 2021年 / 152卷
关键词
Conformal prediction; natural language understanding; paraphrase detection; transformers; CONFIDENCE ESTIMATION;
D O I
暂无
中图分类号
TP18 [人工智能理论];
学科分类号
081104 ; 0812 ; 0835 ; 1405 ;
摘要
Transformer architectures have established themselves as the state-of-the-art in many areas of natural language processing (NLP), including paraphrase detection (PD). However, they do not include a confidence estimation for each prediction and, in many cases, the applied models are poorly calibrated. These features are essential for numerous real-world applications. For example, in those cases when PD is used for sensitive tasks, like plagiarism detection, hate speech recognition or in medical NLP, mistakes might be very costly. In this work we build several variants of transformer-based conformal predictors and study their behaviour on a standard PD dataset. We show that our models are able to produce valid predictions while retaining the accuracy of the original transformer-based models. The proposed technique can be extended to many more NLP problems that are currently being investigated.
引用
收藏
页码:243 / 265
页数:23
相关论文
共 49 条
[1]  
Bahdanau D, 2016, Arxiv, DOI arXiv:1409.0473
[2]  
Beck Daniel, 2016, P 20 SIGNLL C COMP N, P208, DOI [DOI 10.18653/V1/K16-1021, 10.18653/v1/K16-1021]
[3]  
Beltagy I, 2020, Arxiv, DOI arXiv:2004.05150
[4]  
Blatz John, 2004, COLING 2004 P 20 INT, P315
[5]  
Devlin J, 2019, 2019 CONFERENCE OF THE NORTH AMERICAN CHAPTER OF THE ASSOCIATION FOR COMPUTATIONAL LINGUISTICS: HUMAN LANGUAGE TECHNOLOGIES (NAACL HLT 2019), VOL. 1, P4171
[6]  
Dodge J, 2020, Arxiv, DOI [arXiv:2002.06305, 10.48550/arXiv.2002.06305, DOI 10.48550/ARXIV.2002.06305]
[7]  
Dolan WB, 2005, P 3 INT WORKSH PAR I, P9
[8]  
Dong L, 2018, PROCEEDINGS OF THE 56TH ANNUAL MEETING OF THE ASSOCIATION FOR COMPUTATIONAL LINGUISTICS (ACL), VOL 1, P743
[9]  
Falke Tobias, 2020, P 28 INT C COMP LING, P21
[10]  
Fedorova V, 2013, IFIP ADV INF COMM TE, V412, P371