Solving overfitting problems of privacy attacks on small-sample remote sensing data is still a big challenge in practical application. We propose a new privacy attack network, called joint residual network (JRN), for deep learning based privacy objects classification of small-sample remote sensing images in this paper. Unlike the original residual network structure, which add the bottom feature map to top feature map, JRN fuses the bottom feature map with top feature map by matrix joint. It can reduce the possibility that convolution layers extract the noise of training set or consider the inherent attributes of training set as the whole sample attributes. A series benchmark experiments based on GoogleNet model have been enforced and finally, we compare the model process output and the classification accuracy on small-sample data sets. On the UCMLU data set, the GoogleNet-Feat model which is integrated with JRN is 1.66% higher of accuracy than the original GoogleNet model and 1.87% higher than the GoogleNet-R model; on the WHU-RS dataset, GoogleNet-Feat model is 1.04% higher than the GoogleNet model, and is 3.12% higher than the GoogleNet-R model. Compared with the contrast experiments, the classification accuracy of GoogleNet-Feat is the highest when facing the overfitting problems resulting from the small samples.
