KATE: K-Competitive Autoencoder for Text

被引:46
作者
Chen, Yu [1 ]
Zaki, Mohammed J. [1 ]
机构
[1] Rensselaer Polytech Inst, Troy, NY 12180 USA
来源
KDD'17: PROCEEDINGS OF THE 23RD ACM SIGKDD INTERNATIONAL CONFERENCE ON KNOWLEDGE DISCOVERY AND DATA MINING | 2017年
基金
美国国家科学基金会;
关键词
Representation Learning; Autoencoders; Competitive Learning; Text Analytics;
D O I
10.1145/3097983.3098017
中图分类号
TP18 [人工智能理论];
学科分类号
081104 ; 0812 ; 0835 ; 1405 ;
摘要
Autoencoders have been successful in learning meaningful representations from image datasets. However, their performance on text datasets has not been widely studied. Traditional autoencoders tend to learn possibly trivial representations of text documents due to their confounding properties such as high-dimensionality, sparsity and power-law word distributions. In this paper, we propose a novel k-competitive autoencoder, called KATE, for text documents. Due to the competition between the neurons in the hidden layer, each neuron becomes specialized in recognizing specific data patterns, and overall the model can learn meaningful representations of textual data. A comprehensive set of experiments show that KATE can learn better representations than traditional autoencoders including denoising, contractive, variational, and k-sparse autoencoders. Our model also outperforms deep generative models, probabilistic topic models, and even word representation models (e.g., Word2Vec) in terms of several downstream tasks such as document classification, regression, and retrieval.
引用
收藏
页码:85 / 94
页数:10
相关论文
共 45 条
[1]  
[Anonymous], 5 INT C WIK COMM
[2]  
[Anonymous], 2009, Advances in neural information processing systems
[3]  
[Anonymous], 2015, ADV NEURAL INFORM PR
[4]  
[Anonymous], 2011, P 28 INT C MACHINE L
[5]   The Nested Chinese Restaurant Process and Bayesian Nonparametric Inference of Topic Hierarchies [J].
Blei, David M. ;
Griffiths, Thomas L. ;
Jordan, Michael I. .
JOURNAL OF THE ACM, 2010, 57 (02)
[6]   Latent Dirichlet allocation [J].
Blei, DM ;
Ng, AY ;
Jordan, MI .
JOURNAL OF MACHINE LEARNING RESEARCH, 2003, 3 (4-5) :993-1022
[7]  
Blunsom Phil, 2014, ACL M
[8]  
Canini Kevin R., 2009, P INT C ART INT STAT, V5, P65
[9]  
Cao ZQ, 2015, AAAI CONF ARTIF INTE, P2210
[10]  
Das R, 2015, PROCEEDINGS OF THE 53RD ANNUAL MEETING OF THE ASSOCIATION FOR COMPUTATIONAL LINGUISTICS AND THE 7TH INTERNATIONAL JOINT CONFERENCE ON NATURAL LANGUAGE PROCESSING, VOL 1, P795