Hierarchical nucleation in deep neural networks

被引:0
|
作者
Doimo, Diego [1 ]
Glielmo, Aldo [1 ]
Ansuini, Alessio [2 ]
Laio, Alessandro [1 ]
机构
[1] Scuola Int Super Studi Avanzati, Trieste, Italy
[2] Area Sci Pk, Trieste, Italy
来源
ADVANCES IN NEURAL INFORMATION PROCESSING SYSTEMS 33, NEURIPS 2020 | 2020年 / 33卷
关键词
CRITERIA;
D O I
暂无
中图分类号
TP18 [人工智能理论];
学科分类号
081104 ; 0812 ; 0835 ; 1405 ;
摘要
Deep convolutional networks (DCNs) learn meaningful representations where data that share the same abstract characteristics are positioned closer and closer. Understanding these representations and how they are generated is of unquestioned practical and theoretical interest. In this work we study the evolution of the probability density of the ImageNet dataset across the hidden layers in some state-of-the-art DCNs. We find that the initial layers generate a unimodal probability density getting rid of any structure irrelevant for classification. In subsequent layers density peaks arise in a hierarchical fashion that mirrors the semantic hierarchy of the concepts. Density peaks corresponding to single categories appear only close to the output and via a very sharp transition which resembles the nucleation process of a heterogeneous liquid. This process leaves a footprint in the probability density of the output layer where the topography of the peaks allows reconstructing the semantic relationships of the categories.
引用
收藏
页数:11
相关论文
共 50 条
  • [31] Hierarchical Fault Simulation of Deep Neural Networks on Multi-Core Systems
    Karami, Masoomeh
    Haghbayan, Mohammad-Hashem
    Ebrahimi, Masoumeh
    Miele, Antonio
    Tenhunen, Hannu
    Plosila, Juha
    2021 IEEE EUROPEAN TEST SYMPOSIUM (ETS 2021), 2021,
  • [32] Hierarchical committee of deep convolutional neural networks for robust facial expression recognition
    Bo-Kyeong Kim
    Jihyeon Roh
    Suh-Yeon Dong
    Soo-Young Lee
    Journal on Multimodal User Interfaces, 2016, 10 : 173 - 189
  • [33] Extracting Interpretable Hierarchical Rules from Deep Neural Networks' Latent Space
    Wang, Ya
    Paschke, Adrian
    RULES AND REASONING, RULEML+RR 2023, 2023, 14244 : 238 - 253
  • [34] Hierarchical Human-Like Deep Neural Networks for Abstractive Text Summarization
    Yang, Min
    Li, Chengming
    Shen, Ying
    Wu, Qingyao
    Zhao, Zhou
    Chen, Xiaojun
    IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS, 2021, 32 (06) : 2744 - 2757
  • [35] aHCQ: Adaptive Hierarchical Clustering Based Quantization Framework for Deep Neural Networks
    Hu, Jiaxin
    Rao, Weixiong
    Zhao, Qinpei
    ADVANCES IN KNOWLEDGE DISCOVERY AND DATA MINING, PAKDD 2021, PT II, 2021, 12713 : 207 - 218
  • [36] PRoof: A Comprehensive Hierarchical Profiling Framework for Deep Neural Networks with Roofline Analysis
    Wu, Siyu
    Gong, Ruihao
    Yang, Hailong
    Liu, Yi
    Qian, Depei
    You, Xin
    Luan, Zhongzhi
    53RD INTERNATIONAL CONFERENCE ON PARALLEL PROCESSING, ICPP 2024, 2024, : 822 - 832
  • [37] Hierarchical Meta-learning Models with Deep Neural Networks for Spectrum Assignment
    Rutagemwa, Humphrey
    Baddour, Kareem E.
    Rong, Bo
    2019 IEEE INTERNATIONAL CONFERENCE ON COMMUNICATIONS WORKSHOPS (ICC WORKSHOPS), 2019,
  • [38] Single Channel Speech Source Separation Using Hierarchical Deep Neural Networks
    Noorani, Seyed Majid
    Seyedin, Sanaz
    2020 28TH IRANIAN CONFERENCE ON ELECTRICAL ENGINEERING (ICEE), 2020, : 466 - 470
  • [39] Hierarchical committee of deep convolutional neural networks for robust facial expression recognition
    Kim, Bo-Kyeong
    Roh, Jihyeon
    Dong, Suh-Yeon
    Lee, Soo-Young
    JOURNAL ON MULTIMODAL USER INTERFACES, 2016, 10 (02) : 173 - 189
  • [40] Hierarchical Meta-learning Models with Deep Neural Networks for Spectrum Assignment
    Rutagemwa, Humphrey
    Baddour, Kareem E.
    Rong, Bo
    2019 IEEE PACIFIC RIM CONFERENCE ON COMMUNICATIONS, COMPUTERS AND SIGNAL PROCESSING (PACRIM), 2019,