First-principles study of magnetic states and the anomalous Hall conductivity of MNb3S6 (M = Co, Fe, Mn, and Ni)

Inspired by the observation of the extremely large anomalous Hall effect in the absence of applied magnetic fields or uniform magnetization in CoNb3S6 [Nat. Commun. 9, 3280 (2018); Phys. Rev. Research 2, 023051 (2020)], we perform a first-principles study of this and related compounds of the MNb3S6 type with different transition metal M ions to determine their magnetic orders and the anomalous Hall conductivity (AHC). We find that noncoplanar antiferromagnetic ordering is favored relative to collinear or coplanar order in the case of M = Co, Fe, and Ni, while ferromagnetic ordering is favored in MnNb3S6 at low temperatures. The AHC in these materials with noncoplanar spin ordering can reach about e(2)/h per crystalline layer, while being negligible for coplanar and collinear cases. We also find that the AHC depends sensitively on doping and reaches a maximum for intermediate values of the local spin exchange potential between 0.3 and 0.8 eV. Our AHC results are consistent with the reported Hall measurements in CoNb3S6 and suggest a possibility of similarly large anomalous Hall effects in related compounds.