Twisted transition metal dichalcogenide (TMD) homobilayers have recently emerged as a powerful platform for studying correlated insulating states. In the strongly correlated limit, we construct an effective spin Hamiltonian on a honeycomb lattice that includes the Heisenberg interaction and nonsymmetric interactions such as a Dzyaloshinskii-Moriya interaction and a Kane-Mele coupling for the Mott-insulating phase at half-filling. For the twisted TMD homobilayers, the spin-orbit coupling in the Hubbard model, which is expected to induce the antisymmetric exchange couplings in the effective spin Hamiltonian, is a highly tunable and experimentally accessible quantity that can be tuned by an applied electric field. In this study, we investigate classical and quantum phase diagrams of the effective spin Hamiltonian using analytical and numerical methods. We show that the model exhibits a rich classical phase diagram including an antiferromagnetic (AFM) phase, a planar spiral ordered phase with high classical degeneracy, a z-AFM phase, a noncoplanar phase, a noncollinear phase, and a 120 degrees-AFM phase. In the quantum treatment, we calculate low-energy magnon excitation spectrum, ground-state energy, and static spin structure factor using linear spin-wave theory and density matrix renormalization group methods to compose the quantum phase diagram of the effective spin Hamiltonian. Beyond the Heisenberg interaction, we find that the existence of these antisymmetric couplings is responsible for the quantum spin liquid, z-AFM, noncoplanar, and 120 degrees phases. Twisted TMD homobilayers, therefore, offer rich platforms for realizing rich phases of matter such as quantum spin liquid, noncoplanar, and 120 degrees, resulting from the spin-orbit coupling.
