Inclination instability of circumbinary planets

We analyse a tilt instability of the orbit of an outer planet in a two-planet circumbinary system that we recently reported. The binary is on an eccentric orbit and the inner circumbinary planet is on a circular polar orbit that causes the binary to undergo apsidal precession. The outer circumbinary planet is initially on a circular or eccentric orbit that is coplanar with respect to the binary. We apply a Hamiltonian in quadrupole order of the binary potential to show that the tilt instability is the result of a secular resonance in which the apsidal precession rate of the binary matches the nodal precession rate of the outer planet. A resonance is possible because the polar inner planet causes the apsidal precession of the binary to be retrograde. The outer planet periodically undergoes large tilt oscillations for which we analytically determine the initial evolution and maximum inclination. Following a typically relatively short adjustment phase, the tilt grows exponentially in time at a characteristic rate that is of order the absolute value of the binary apsidal precession rate. The analytic results agree well with numerical simulations. This instability is analogous to the Kozai-Lidov instability, but applied to a circumbinary object. The instability fails to operate if the binary mass ratio is too extreme. The instability occurs even if the outer planet is instead an object of stellar mass and involves tilt oscillations of the inner binary.