Interacting spin-3/2 fermions in a Luttinger semimetal: Competing phases and their selection in the global phase diagram

We compute the effects of electronic interactions on gapless spin-3/2 excitations that in a noninteracting system emerge at a biquadratic touching of Kramers degenerate valence and conduction bands in three dimensions, also known as a Luttinger semimetal. This model can describe the low-energy physics of HgTe, gray-Sn, 227 pyrochlore iridates, and half-Heuslers. For the sake of concreteness, we only consider the short-range components of the Coulomb interaction (extended Hubbard-like model). By combining mean-field analysis with a renormalization group (RG) calculation (controlled by a "small" parameter epsilon, where epsilon = d - 2), we construct multiple cuts of the global phase diagram of interacting spin-3/2 fermions at zero and finite temperature and chemical doping. Such phase diagrams display a rich confluence of competing orders, among which rotational symmetry breaking nematic insulators and time-reversal symmetry breaking magnetic orders (supporting Weyl quasiparticles) are the prominent candidates for excitonic phases. We also show that even repulsive interactions can be conducive to both mundane s-wave and topological d-wave pairings. The reconstructed band structure (within the mean-field approximation) inside the ordered phases allows us to organize them according to the energy (entropy) gain in the following (reverse) order: s-wave pairing, nematic phases, magnetic orders, and d-wave pairings, at zero chemical doping. However, the paired states are energetically superior over the excitonic ones for finite doping. The phase diagrams obtained from the RG analysis show that for sufficiently strong interactions, an ordered phase with higher energy (entropy) gain is realized at low (high) temperature. In addition, we establish a "selection rule" between the interaction channels and the resulting ordered phases, suggesting that repulsive short-range interactions in the magnetic (nematic) channels are conducive to the nucleation of d-wave (s-wave) pairing among spin-3/2 fermions. We believe that the proposed methodology can shed light on the global phase diagram of various two and three-dimensional interacting multiband systems, such as Dirac materials, doped topological insulators, and the like.