Context Vectors Are Reflections of Word Vectors in Half the Dimensions

被引:8
作者
Assylbekov, Zhenisbek [1 ]
Takhanov, Rustem [1 ]
机构
[1] Nazarbayev Univ, Dept Math, 53 Kabanbay Batyr Ave, Astana 010000, Kazakhstan
关键词
Embeddings;
D O I
10.1613/jair.1.11368
中图分类号
TP18 [人工智能理论];
学科分类号
081104 ; 0812 ; 0835 ; 1405 ;
摘要
This paper takes a step towards the theoretical analysis of the relationship between word embeddings and context embeddings in models such as word2vec. We start from basic probabilistic assumptions on the nature of word vectors, context vectors, and text generation. These assumptions are supported either empirically or theoretically by the existing literature. Next, we show that under these assumptions the widely-used word-word PMI matrix is approximately a random symmetric Gaussian ensemble. This, in turn, implies that context vectors are reflections of word vectors in approximately half the dimensions. As a direct application of our result, we suggest a theoretically grounded way of tying weights in the SGNS model.
引用
收藏
页码:225 / 242
页数:18
相关论文
共 34 条
[1]  
Allen C., 2018, ARXIV180512164
[2]  
Andreas J., 2015, P 2015 C N AM CHAPT, P244
[3]  
[Anonymous], P ICLR
[4]  
[Anonymous], P ICLR
[5]  
[Anonymous], 2015, Transactions of the Association for Computational Linguistics, DOI DOI 10.1186/1472-6947-15-S2-S2.ARXIV:1103.0398
[6]  
Arora Sanjeev, 2016, Trans. Assoc. Comput. Linguistics, V4, P385
[7]  
Bengio Y., 2001, A neural probabilistic language model
[8]  
Bruni L, 2012, GENESIS AND ETHOS OF THE MARKET, P136
[9]   The small world of human language [J].
Cancho, RFI ;
Solé, RV .
PROCEEDINGS OF THE ROYAL SOCIETY B-BIOLOGICAL SCIENCES, 2001, 268 (1482) :2261-2265
[10]  
De Monvel AB, 1999, ANN PROBAB, V27, P913