Antiferromagnetic ground state with pair-checkerboard order in FeSe

A monolayer FeSe thin film grown on SrTiO3(001) (STO) shows the sign of T-c > 77 K, which is higher than the Tc record of 56 K for bulk FeAs-based superconductors. However, little is known about the magnetic ground state of FeSe, which should be closely related to its unusual superconductivity. Previous studies presume the collinear stripe antiferromagnetic (AFM) state as the ground state of FeSe, the same as that in FeAs superconductors. Here we find a magnetic order named the "pair-checkerboard AFM" as the magnetic ground state of tetragonal FeSe. The pair-checkerboard order results from the interplay between the nearest-, next-nearest, and unnegligible next-next-nearest neighbor magnetic exchange couplings of Fe atoms. The monolayer FeSe in pair-checkerboard order shows an unexpected insulating behavior with a Dirac-cone-like band structure related to the specific orbital order of the d(xz) and d(yz) characters of Fe atoms, which could explain the recently observed insulator-superconductor transition. The present results cast insights on the magnetic ordering in FeSe monolayer and its derived superconductors.