Self-compensation in Ge- and Si-doped Al0.3Ga0.7N has been investigated in terms of the formation of III vacancy and donor-vacancy complexes. Both Ge- and Si-doped AlGaN layers showed a compensation knee behavior with impurity compensation (low doping regime), compensation plateau (medium doping regime), and self-compensation (high doping regime). A maximum free carrier concentration of 4-5x10(19)cm(-3) was obtained by Ge doping, whereas Si doping resulted in only half of that value, similar to 2x10(19)cm(-3). A DFT calculation with the grand canonical thermodynamics model was developed to support the hypothesis that the difference in self-compensation arises from the difference in the formation energies of the V-III-n center dot donor complexes relative to their onsite configurations. The model suggested that the V-III-2 center dot donor and V-III-3 center dot donor complexes were responsible for self-compensation for both Ge- and Si-doped AlGaN. However, a lower free carrier concentration in Si-doped samples was due to a high V-III-3 center dot Si concentration, resulting from a lower energy of formation of V-III-3 center dot Si.