Brain-inspired Large-scale Deep Neural Network System

被引：0
|
作者：
Lü J.-C. [1 ]
Ye Q. [1 ]
Tian Y.-X. [1 ]
Han J.-W. [2 ]
Wu F. [3 ]
机构：
[1] College of Computer Science, Sichuan University, Chengdu
[2] School of Automation, Northwestern Polytechnical University, Xi'an
[3] Department of Electronic Engineering and Information Science, University of Science and Technology of China, Hefei
来源：
Ruan Jian Xue Bao/Journal of Software | 2022年 / 33卷 / 04期
关键词：
Brain science; Distributed computing; General artificial intelligence; Large-scale deep neural networks; Multi-modal;
D O I：
10.13328/j.cnki.jos.006470
中图分类号：
学科分类号：
摘要：
Large-scale deep neural networks (DNNs) exhibit powerful end-to-end representation and infinite approximation of nonlinear functions, showing excellent performance in several fields and becoming an important development direction. For example, the natural language processing model GPT, after years of development, now has 175 billion network parameters and achieves state-of-the-art performance on several NLP benchmarks. However, according to the existing deep neural network organization, the current large-scale network is difficult to reach the scale of human brain biological neural network connection. At the same time, the existing large-scale DNNs do not perform well in multi-channel collaborative processing, knowledge storage, and reasoning. This study proposes a brain-inspired large-scale DNN model, which is inspired by the division and the functional mechanism of brain regions and built modularly by the functional of the brain, integrates a large amount of existing data and pre-trained models, and proposes the corresponding learning algorithm by the functional mechanism of the brain. The DNN model implements a pathway to automatically build a DNN as an output using the scene as an input. Simultaneously, it should not only learn the correlation between input and output but also needs to have the multi-channel collaborative processing capability to improve the correlation quality, thereby realizing knowledge storage and reasoning ability, which could be treated as a way toward general artificial intelligence. The whole model and all data sets and brain-inspired functional areas are managed by a database system which is equipped with the distributed training algorithms to support the efficient training of the large-scale DNN on computing clusters. This study also proposes a novel idea to implement general artificial intelligence, and the large-scale model is validated on several different modal tasks. © Copyright 2022, Institute of Software, the Chinese Academy of Sciences. All rights reserved.
引用
收藏
页码：1412 / 1429
页数：17
相关论文
共 45 条
  • [1] He K, Zhang X, Ren S, Et al., Deep residual learning for image recognition, Proc. of the 2016 IEEE Conf. on Computer Vision and Pattern Recognition, pp. 770-778, (2016)
  • [2] Krizhevsky A, Sutskever I, Hinton G., ImageNet classification with deep convolutional neural networks, Advances in Neural Information Processing Systems, 25, pp. 1106-1114, (2012)
  • [3] Kingsbury B., Deep neural networks for acoustic modeling in speech recognition: The shared views of four research groups, IEEE Signal Processing Magazine, 29, 6, pp. 82-97, (2012)
  • [4] Miao Y, Gowayyed M, Metze F., EESEN: End-to-end speech recognition using deep RNN models and wfst-based decoding, Proc. of the 2015 IEEE Workshop on Automatic Speech Recognition and Understanding, pp. 167-174, (2015)
  • [5] Brown TB, Mann B, Ryder N, Et al., Language models are few-shot learners, Advances in Neural Information Processing Systems, (2020)
  • [6] Brock A, Donahue J, Simonyan K., Large scale gan training for high fidelity natural image synthesis, Proc. of the 7th Int'l Conf. on Learning Representations, (2019)
  • [7] Devlin J, Chang M-W, Lee K, Et al., Bert: Pre-training of deep bidirectional transformers for language understanding, Proc. of the 2019 Conf. of the North American Chapter of the Association for Computational Linguistics, pp. 4171-4186, (2019)
  • [8] Hu J, Shen L, Sun G., Squeeze-and-excitation networks, Proc. of the 2018 IEEE Conf. on Computer Vision and Pattern Recognition, pp. 7132-7141, (2018)
  • [9] Chen YJ, Huang TJ, Pan G, Et al., Research progress and development trend of brain-like computing, China Computer Federation Proceedings CCFP 0027, pp. 263-317, (2016)
  • [10] Yang GW, Wang SJ, Yan QX., Research of fractional linear neural network and its ability for nonlinear approach, Chinese Journal Of Computers, 30, 2, pp. 189-199, (2007)
12345