Orbital nematic order and interplay with magnetism in the two-orbital Hubbard model

Motivated by the recent angle-resolved photoemission spectroscopy (ARPES) on FeSe and iron pnictide families of iron-based superconductors, we have studied the orbital nematic order and its interplay with antiferromagnetism within the two-orbital Hubbard model. We used random phase approximation (RPA) to calculate the dependence of the orbital and magnetic susceptibilities on the strength of interactions and electron density (doping). To account for strong electron correlations not captured by RPA, we further employed non-perturbative variational cluster approximation (VCA) capable of capturing symmetry broken magnetic and orbitally ordered phases. Both approaches show that the electron and hole doping affect the two orders differently. While hole doping tends to suppress both magnetism and orbital ordering, the electron doping suppresses magnetism faster. Crucially, we find a realistic parameter regime for moderate electron doping that stabilizes orbital nematicity in the absence of long-range antiferromagnetic order. This is reminiscent of the non-magnetic orbital nematic phase observed recently in FeSe and a number of iron pnictide materials and raises the possibility that at least in some cases, the observed electronic nematicity may be primarily due to orbital rather than magnetic fluctuations.