The accurate matching of planetary remote sensing images (PRSIs) is the premise of accurate planetary terrain mapping. However, PRSIs often lack apparent man-made structures such as buildings or roads, leading to difficulties in feature description. In addition, the PRSIs collected by different sensors are affected by the imaging mechanism and the solar illumination, and there are obvious nonlinear radiation differences (NRDs). These problems make the matching of PRSIs difficult. To address the above issues, this article proposes a PRSI matching method based on optimized bandpass filtering and local topological prior, divided into two stages: coarse matching and fine matching. In the coarse matching stage, we first use the bandpass filtering to calculate the phase congruency (PC). Then, the feature block descriptors are constructed, and the local topology consensus is used to achieve the coarse alignment of feature blocks. Finally, we extract the point features and use the matching results of block features to narrow the matching range of point features. Based on the coarse matching results, the precision and reliability of the results are further improved through fine matching. The experimental results achieved with a PRSI dataset with 75 image pairs demonstrate that our method is superior to other recent methods, the matching accuracy of the proposed method is improved by more than 2.367 pixels, and the success rate is improved by over 22.667%. The source code will be publicly available at https://github.com/WGY-RS/OFLP.