The instabilities in rotating detonation invoke potentially adverse influences on the operation stability. Hishida et al. first found the rippled structure existed in the interface between fresh injections and burnt products from the previous cycle (Hishida et al., Shock Waves, 19, 1(2009)), and they suggested a mechanism of Kelvin-Helmholtz instability as well. In order to achieve a deep understanding on this issue, simulations are performed by using a fifth-order WENO-type scheme with improved resolution and 7-species-and-8-reaction chemical model regarding H-2/air mixture; besides, three grids with increasing resolution are employed. The results show that besides the previously-mentioned Kelvin-Helmholtz instability, there are two other mechanisms which take effect in the interface instability, i.e., the effect of baroclinic torque and Rayleigh-Taylor instability. Occurrence conditions for instabilities are checked and testified. Especially, the spike- and bubble-like structures are observed at the interface and show appearances different from canonical structures by Kelvin-Helmholtz instability. (C) 2018 Elsevier Ltd. All rights reserved.