Feature matching is a crucial and challenging process in feature-based image registration. Mismatch is always inevitable in image registration for the feature matching methods that just use local features, no matter how powerful the discrimination of the feature point descriptor is. To solve this problem, in this paper, relative moment affine invariants are used to compare the similarity of two triangles, then a new global feature matching method is proposed to match the feature points accurately based on graph structure. In the point matching process, Genetic Algorithm is applied to find two most similar graphs that are constructed by the corresponding survivor points from two images. The proposed algorithm can deal with images of affine transformation, large scale and low overlap. Compared with traditional Iterative Closest Point (ICP), normalized cross-correlation (NCC) and Coherent Point Drift (CPD), which register aerial images captured on the sea, the proposed algorithm works well with high accuracy and stability even when the point sets have a lot of outliers.