The standard extended Kalman filter (EKF) model, employed in integrated global navigation satellite systems and strap-down inertial navigation systems (GNSS/SINS), encounters challenges related to inconsistent reference frame definitions attributed to large attitude misalignment, and it grapples with inconsistent variance estimation originating from an inaccurate inertial measuring unit (IMU) measurement. These difficulties are especially notable when applied to Rotor unmanned aerial vehicles (UAVs). This article derives the tightly-coupled (TC) SINS and GNSS real-time kinematic (RTK) measurement models with two invariant error state formulations, namely, the left-invariant EKF which defines attitude, velocity, and position state errors on matrix Lie group (LG), denoted as LG (R,v,p)-LIEKF; and the right-invariant EKF which formulates attitude and velocity state errors as matrix Lie group, denoted as LG (R,v)-RIEKF. Then we conduct an analysis of the characteristics and applicability of the two filtering models and propose a method for seamlessly transitioning between them. Subsequently, we propose a novel framework for the rotor UAV TC RTK/SINS positioning method based on an adaptive invariant EKF (AIEKF). Field flight experiments with a low-cost GNSS/SINS-integrated navigation system on a rotor UAV demonstrate that LG (R,v,p)-LIEK exhibits rapid convergence and is capable of performing integrated navigation even in cases of substantial attitude misalignment; LG (R,v)-RIEKF proves to be effective in reducing the adverse influences of inconsistent variance estimation caused by high-frequency vibrations in UAV. The proposed novel AIEKF framework offers considerable advantages over the standard EKF model, with respect to UAV-integrated positioning accuracy, RTK ambiguity resolution (AR), and computational efficiency.