Visually Analyzing Contextualized Embeddings

被引:7
作者
Berger, Matthew [1 ]
机构
[1] Vanderbilt Univ, 221 Kirkland Hall, Nashville, TN 37235 USA
来源
2020 IEEE VISUALIZATION CONFERENCE - SHORT PAPERS (VIS 2020) | 2020年
关键词
Machine Learning visual analytics; Natural Language Processing;
D O I
10.1109/VIS47514.2020.00062
中图分类号
TP31 [计算机软件];
学科分类号
081202 ; 0835 ;
摘要
In this paper we introduce a method for visually analyzing contextualized embeddings produced by deep neural network-based language models. Our approach is inspired by linguistic probes for natural language processing, where tasks are designed to probe language models for linguistic structure, such as parts-of-speech and named entities. These approaches are largely confirmatory, however, only enabling a user to test for information known a priori. In this work, we eschew supervised probing tasks, and advocate for unsupervised probes, coupled with visual exploration techniques, to assess what is learned by language models. Specifically, we cluster contextualized embeddings produced from a large text corpus, and introduce a visualization design based on this clustering and textual structure - cluster co-occurrences, cluster spans, and cluster-word membership - to help elicit the functionality of, and relationship between, individual clusters. User feedback highlights the benefits of our design in discovering different types of linguistic structures.
引用
收藏
页码:276 / 280
页数:5
相关论文
共 37 条
[1]  
Arthur D, 2007, PROCEEDINGS OF THE EIGHTEENTH ANNUAL ACM-SIAM SYMPOSIUM ON DISCRETE ALGORITHMS, P1027
[2]   Analysis Methods in Neural Language Processing: A Survey [J].
Belinkov, Yonatan ;
Glass, James .
TRANSACTIONS OF THE ASSOCIATION FOR COMPUTATIONAL LINGUISTICS, 2019, 7 :49-72
[3]   A neural probabilistic language model [J].
Bengio, Y ;
Ducharme, R ;
Vincent, P ;
Jauvin, C .
JOURNAL OF MACHINE LEARNING RESEARCH, 2003, 3 (06) :1137-1155
[4]  
Bjerva J., 2016, ARXIV PREPRINT ARXIV
[5]   Embedding Comparator: Visualizing Differences in Global Structure and Local Neighborhoods via Small Multiples [J].
Boggust, Angie ;
Carter, Brandon ;
Satyanarayan, Arvind .
IUI'22: 27TH INTERNATIONAL CONFERENCE ON INTELLIGENT USER INTERFACES, 2022, :746-766
[6]  
Cheonbok Park, 2019, 2019 IEEE Visualization Conference (VIS), P146, DOI 10.1109/VISUAL.2019.8933677
[7]   What does BERT look at? An Analysis of BERT's Attention [J].
Clark, Kevin ;
Khandelwal, Urvashi ;
Levy, Omer ;
Manning, Christopher D. .
BLACKBOXNLP WORKSHOP ON ANALYZING AND INTERPRETING NEURAL NETWORKS FOR NLP AT ACL 2019, 2019, :276-286
[8]  
Conneau A, 2018, PROCEEDINGS OF THE 56TH ANNUAL MEETING OF THE ASSOCIATION FOR COMPUTATIONAL LINGUISTICS (ACL), VOL 1, P2126
[9]  
Devlin J, 2019, 2019 CONFERENCE OF THE NORTH AMERICAN CHAPTER OF THE ASSOCIATION FOR COMPUTATIONAL LINGUISTICS: HUMAN LANGUAGE TECHNOLOGIES (NAACL HLT 2019), VOL. 1, P4171
[10]  
Gardner M, 2020, ARXIV PREPRINT ARXIV