Self-Supervised Encoders Are Better Transfer Learners in Remote Sensing Applications

Transfer learning has been shown to be an effective method for achieving high-performance models when applying deep learning to remote sensing data. Recent research has demonstrated that representations learned through self-supervision transfer better than representations learned on supervised classification tasks. However, little research has focused explicitly on applying self-supervised encoders to remote sensing tasks. Using three diverse remote sensing datasets, we compared the performance of encoders pre-trained through both supervision and self-supervision on ImageNet, then fine-tuned on a final remote sensing task. Furthermore, we explored whether performance benefited from further pre-training on remote sensing data. Our experiments used SwAV due to its comparably lower computational requirements, as this method would prove most easily replicable by practitioners. We show that an encoder pre-trained on ImageNet using self-supervision transfers better than one pre-trained using supervision on three diverse remote sensing applications. Moreover, self-supervision on the target data alone as a pre-training step seldom boosts performance beyond this transferred encoder. We attribute this inefficacy to the lower diversity and size of remote sensing datasets, compared to ImageNet. In conclusion, we recommend that researchers use self-supervised representations for transfer learning on remote sensing data and that future research should focus on ways to increase performance further using self-supervision.