Magnetic ordering phenomena of interacting quantum spin Hall models

The two-dimensional Hubbard model defined for topological band structures exhibiting a quantum spin Hall effect poses fundamental challenges in terms of phenomenological characterization and microscopic classification. In the limit of infinite coupling U at half filling, the spin model Hamiltonians resulting from a strong-coupling expansion show various forms of magnetic ordering phenomena depending on the underlying spin-orbit coupling terms. We investigate the infinite-U limit of the Kane-Mele-Hubbard model with z-axis intrinsic spin-orbit coupling as well as its generalization to a generically multidirectional spin-orbit term which has been claimed to account for the physical scenario in monolayer Na2IrO3. We find that the axial spin symmetry which is kept in the former but broken in the latter has a fundamental impact on the magnetic phase diagram as we vary the spin-orbit coupling strength. While the Kane-Mele spin model shows a continuous evolution from conventional honeycomb Neel to XY antiferromagnetism which avoids the frustration imposed by the increased spin-orbit coupling, the multidirectional spin-orbit term induces a commensurate to incommensurate transition at intermediate coupling strength, and yields a complex spiral state with a 24 site unit cell in the limit of infinite spin-orbit coupling. From our findings, we conjecture that in the case of broken axial spin symmetry there is a large propensity for an additional phase at sufficiently large spin-orbit coupling and intermediate U.