The heliopause (HP) separates the tenuous hot heliosheath plasma from the relatively dense cool magnetized plasma of the local interstellar medium (LISM). Fluid acceleration in the HP region can therefore drive Rayleigh-Taylor-like and Kelvin-Helmholtz-like instabilities. Charge exchange coupling of plasma ions and primary interstellar neutral atoms provides an effective gravity, suggesting the possibility of Rayleigh Taylor-like (RT-like) instabilities. Shear flow due to the velocity difference between the heliosheath and the interstellar flows drives Kelvin Helmholtz-like (KH-like) modes on the heliopause. Magnetic fields damp the classical KH instability. However, we show that energetic neutral atoms (ENAs) destabilize KH-modes,even in the presence of interplanetary and interstellar magnetic fields. We consider a model that includes a number of effects that are important in the heliosphere such as resonant change exchange between the primary neutrals and the solar wind plasma, ENAs from the inner heliosheath, plasma flows along the heliopause and magnetic fields in the inner and outer heliosheath. We find that the nose region is unstable to RT-like modes for HP parameters, while the shoulder region is unstable to a new instability that has the characteristics of a mixed RT-KH-like mode. These instabilities are not stabilized by typical values of the magnetic fields in the inner and outer heliosheath close to the nose and shoulder regions. Whereas ENAs have a stabilizing influence on the RT instability in the vicinity of the nose region (due to counter streaming), they have a destabilizing influence on the KH instability in the vicinity of the flanks. We find that even in the presence of interplanetary and interstellar magnetic fields, ENAs can drive a new form of KH-like instability on the flanks. An analysis of the collisional and anomalous magnetic field diffusion time scales shows that ideal MHD is an appropriate model at the HP. The interstellar magnetic field therefore drapes over the HP and does not diffuse into the inner heliosheath (IHS). However, RT-like, RT-KH-like, and KH-like instabilities serve to drag outer heliosheath (OHS)/interstellar magnetic field into the IHS, allowing for local reconnection of interplanetary and interstellar magnetic field. Such reconnection may 1) enhance the mixing of plasmas across the heliopause, and 2) provide open magnetic field lines that allow easy ingress of galactic cosmic rays into the heliosphere and easy loss of anomalous cosmic rays.
