The linear instability of circular vortices over isolated topography in a homogeneous and inviscid fluid is examined for the shallow-water and quasi-geostrophic models in the f-plane. The eigenvalue problem associated with azimuthal disturbances is derived for arbitrary axisymmetric topographies, either submarine mountains or valleys. Amended Rayleigh and Fjortoft theorems with topographic effects are given for barotropic instability, obtaining necessary criteria for instability when the potential vorticity gradient is zero somewhere in the domain. The onset of centrifugal instability is also discussed by deriving the Rayleigh circulation theorem with topography. The barotropic instability theorems are applied to a wide family of nonlinear, quasi-geostrophic solutions of circular vortices over axisymmetric topographic features. Flow instability depends mainly on the vortex/topography configuration, as well as on the vortex size in comparison with the width of the topography. It is found that anticyclones/mountains and cyclones/valleys may be unstable. In contrast, cyclone/mountain and anticyclone/valley configurations are stable. These statements are validated with two numerical methods. First, the generalised eigenvalue problem is solved to obtain the wavenumber of the fastest-growing perturbations. Second, the evolution of the vortices is simulated numerically to detect the development of linear perturbations. The numerical results show that for unstable vortices over narrow topographies, the fastest growth rate corresponds to mode 1, which subsequently forms asymmetric dipolar structures. Over wide topographies, the fastest perturbations are mainly modes 1 and 2, depending on the topographic features.
