Repeatable adaptive keypoint detection via self-supervised learning

Keypoint-based matching is a fundamental technology for different computer vision tasks, in which keypoint detection is a crucial step and directly affects the entire performance. Based on deep learning approaches, the learning-based keypoint detectors have been significantly developed. To further improve the accuracy of high-level matching tasks, the extracted keypoints should provide more accurate point-to-point correspondences and maintain a uniform spatial distribution. Based on this idea, a self-supervised learning method of keypoint detection named repeatable adaptive point is proposed. This method consists of a self-supervised objective and an optimization algorithm. The objective maximizes the repeatability measure with the sparsity constraint of keypoints. This sparsity constraint is formulated by combining the non-maximum suppression operation and the penalty function of keypoint number, which generally makes keypoints have a uniform spatial distribution. A novel approximate alternate optimization algorithm is proposed to maximize the above objective, whose convergence is proved in theory. The proposed detector is “adaptive” because the combinations of it and some existing descriptors can adapt to high-level matching tasks with a fast convergence speed. Specifically, the combinations of it and SuperPoint/HardNet descriptors achieve state-of-the-art accuracy on three high-level tasks based on image matching, namely homography estimation, camera pose estimation, and three-dimensional reconstruction. Furthermore, the proposed method converges faster on new scenes compared with the state-of-the-art method that jointly optimizes the detector and the descriptor.