Anomalous and anisotropic nonlinear susceptibility in the proximate Kitaev magnet α-RuCl3

The leading order nonlinear (NL) susceptibility, chi(3), in a paramagnet is negative and diverges as T -> 0. This divergence is destroyed when spins correlate and the NL response provides unique insights into magnetic order. Dimensionality, exchange interaction, and preponderance of quantum effects all imprint their signatures in the NL magnetic response. Here, we study the NL susceptibilities in the proximate Kitaev magnet alpha-RuCl3, which differs from the expected antiferromagnetic behavior. For T < T-c = 7.5 K and field B in the ab-plane, we obtain contrasting NL responses in low (<2 T) and high field regions. For low fields, the NL behavior is dominated by a quadratic response (positive chi(2)), which shows a rapid rise below T-c. This large chi(2) > 0 implies a broken sublattice symmetry of magnetic order at low temperatures. Classical Monte Carlo (CMC) simulations in the standard K - H - Gamma model secure such a quadratic B dependence of M, only for T approximate to T-c with chi(2) being zero as T -> 0. It is also zero for all temperatures in exact diagonalization calculations. On the other hand, we find an exclusive cubic term (chi(3)) that describes the high field NL behavior well. chi(3) is large and positive both below and above T-c crossing zero only for T > 50 K. In contrast, for B parallel to c-axis, no separate low/high field behaviors are measured and only a much smaller chi(3) is apparent.