Mutual information-based dropout: Learning deep relevant feature representation architectures

被引:9
|
作者
Chen, Jie [1 ,2 ]
Wu, ZhongCheng [1 ,3 ]
Zhang, Jun [1 ]
Li, Fang [1 ]
机构
[1] Chinese Acad Sci, Hefei Inst Phys Sci, Hefei, Anhui, Peoples R China
[2] Univ Sci & Technol China, Grad Sch Comp Appl Technol, Hefei, Anhui, Peoples R China
[3] Univ Sci & Technol China, Hefei, Anhui, Peoples R China
关键词
Mutual information; Feature representation; Dropout; Overfitting; Autoencoder; Convolutional neural networks; NEURAL-NETWORKS; AUTOENCODERS; SELECTION;
D O I
10.1016/j.neucom.2019.04.090
中图分类号
TP18 [人工智能理论];
学科分类号
081104 ; 0812 ; 0835 ; 1405 ;
摘要
We propose a new regularization strategy called DropMI, which is a generalization of Dropout for the regularization of networks that introduces mutual information (MI) dynamic analysis. The standard Dropout randomly drops a certain proportion of neural units, according to the Bernoulli distribution, thereby resulting in the loss of some important hidden feature information. In DropMI, we first evaluate the importance of each neural unit in the feature representation of the hidden layer based on the MI between it and the target. We then construct a new binary mask matrix based on the sorting distribution of MI, thus developing a dynamic DropMI strategy that highlights the important neural units that are beneficial to the feature representation. The results from the MNIST, NORB, CIFAR-10, CIFAR-100, SVHN, and Multi-PIE datasets indicate that, relative to other state-of-the-art regularization methods based on the benchmark autoencoder and convolutional neural networks, our method has better feature representation performance and effectively reduces the overfitting of the model. (C) 2019 Elsevier B.V. All rights reserved.
引用
收藏
页码:173 / 184
页数:12
相关论文
共 50 条
  • [41] Mutual Information-Driven Subject-Invariant and Class-Relevant Deep Representation Learning in BCI
    Jeon, Eunjin
    Ko, Wonjun
    Yoon, Jee Seok
    Suk, Heung-Il
    IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS, 2023, 34 (02) : 739 - 749
  • [42] Mutual information-based context quantization
    Cagnazzo, Marco
    Antonini, Marc
    Barlaud, Michel
    SIGNAL PROCESSING-IMAGE COMMUNICATION, 2010, 25 (01) : 64 - 74
  • [43] Mutual Information-Based Visual Servoing
    Dame, Amaury
    Marchand, Eric
    IEEE TRANSACTIONS ON ROBOTICS, 2011, 27 (05) : 958 - 969
  • [44] Mutual information-based multi-output tree learning algorithm
    Kang, Hyun-Seok
    Jun, Chi-Hyuck
    INTELLIGENT DATA ANALYSIS, 2021, 25 (06) : 1525 - 1545
  • [45] A Mutual Information-Based Assessment of Reverse Engineering on Rewards of Reinforcement Learning
    Chen T.
    Liu J.
    Baker T.
    Wu Y.
    Xiang Y.
    Li Y.
    Niu W.
    Tong E.
    Zomaya A.Y.
    IEEE Transactions on Artificial Intelligence, 2023, 4 (05): : 1089 - 1100
  • [46] Information-Based Disentangled Representation Learning for Unsupervised MR Harmonization
    Zuo, Lianrui
    Dewey, Blake E.
    Carass, Aaron
    Liu, Yihao
    He, Yufan
    Calabresi, Peter A.
    Prince, Jerry L.
    INFORMATION PROCESSING IN MEDICAL IMAGING, IPMI 2021, 2021, 12729 : 346 - 359
  • [47] A Mutual Information-Based Hybrid Feature Selection Method for Software Cost Estimation Using Feature Clustering
    Shi, Shihai
    Liu, Qin
    INTERNATIONAL JOINT CONFERENCE ON APPLIED MATHEMATICS, STATISTICS AND PUBLIC ADMINISTRATION (AMSPA 2014), 2014, : 481 - 490
  • [48] MINT: Deep Network Compression via Mutual Information-based Neuron Trimming
    Ganesh, Madan Ravi
    Corso, Jason J.
    Sekeh, Salimeh Yasaei
    2020 25TH INTERNATIONAL CONFERENCE ON PATTERN RECOGNITION (ICPR), 2021, : 8251 - 8258
  • [49] Mutual information-based filter hybrid feature selection method for medical datasets using feature clustering
    Sadegh Asghari
    Hossein Nematzadeh
    Ebrahim Akbari
    Homayun Motameni
    Multimedia Tools and Applications, 2023, 82 : 42617 - 42639
  • [50] Mutual information-based filter hybrid feature selection method for medical datasets using feature clustering
    Asghari, Sadegh
    Nematzadeh, Hossein
    Akbari, Ebrahim
    Motameni, Homayun
    MULTIMEDIA TOOLS AND APPLICATIONS, 2023, 82 (27) : 42617 - 42639