Ground state phase diagram and the exotic phases in spin-1 2 square lattice J1-J2-Jχ model

The intricate interplay between frustration and spin chirality, induced by the Dzyaloshinskii-Moriya interaction, holds promise for unveiling novel phases in frustrated quantum magnets. This study investigates the ground state phase diagram of the spin- 1 2 square lattice J1-J2 model upon incorporation of the chiral interaction J chi. Employing exact diagonalization (ED) with full lattice symmetries, we analyze the phase evolution as a function of J2 at fixed J chi, utilizing highly symmetric energy levels. Critical level crossings and ground state fidelity susceptibility (FS) techniques aid in pinpointing phase boundaries: Magnetic to nonmagnetic phase transitions are identified through critical level crossings between gapless magnetic excitations and quasidegenerate ground states of nonmagnetic phases. Direct transitions between two nonmagnetic phases are characterized by FS peaks due to avoided ground state level crossing. Based on these observations, we identify an anticipated chiral spin liquid (CSL) state and an adjacent nematic spin liquid (NSL) phase within a significant range of J chi, demarcated by a nearly vertical boundary line at J2 approximate to 0.65. This critical line terminates at the lower boundary in J chi of a magnetic ordered chiral spin solid phase, which gains prominence with increasing J chi from both CSL and NSL phases. The topological nature of the CSL is confirmed using the modular S matrix of minimally entangled states on a torus and the entanglement spectra of even and odd sectors on a cylinder, employing ED and SU(2 )-symmetric density matrix renormalization group method, respectively. Furthermore, a comprehensive discussion on the nature of the NSL is provided, exploring aspects such as ground state degeneracy, local bond energy landscape, and singlet and triplet gaps on various tori, offering substantial evidence supporting the nematic nature of the NSL.