A Novel Connectivity of Deep Convolutional Neural Networks

Residual network(ResNet) is an effective instance and a significant extension of deep convolutional neural network. ResNet utilizes skip-connection between input layers and output layers to solve the vanishing gradient problem. Due to the powerfulness of skip-connection, the gradient can flow directly through the identity function from later layers to the earlier layers. However, skip-connection makes ResNet overemphasizes the effect of the original input feature. In this paper, we further research the learning behavior of deep residual networks(ResNet) and proposed a novel connectivity pattern to replace skip-connection, which builds upon the success of interpreting the ResNet as an exponential ensemble of relatively shallow networks. The proposed method is implemented by making more input layers link with the output layer. The proposed method is called spare residual network(SRN). it generates model that is wider rather than deeper. The experimental results show that the proposed SRN can achieve impressive results on CIFAR-10, and outperform most of the existing architecture.