Population inference of spin-induced quadrupole moments as a probe for nonblack hole compact binaries

Gravitational wave (GW) measurements of physical effects such as spin-induced quadrupole moments can distinguish binaries consisting of black holes from nonblack hole binaries. While these effects may be poorly constrained for single-event inferences with the second-generation detectors, combining information from multiple detections can help uncover features of nonblack hole binaries. The spin-induced quadrupole moment has specific predictions for different types of compact objects, and a generalized formalism must consider a population where different types of compact objects co-exist. In this study, we introduce a hierarchical mixture-likelihood formalism to estimate the fraction of nonbinary black holes in the population. We demonstrate the applicability of this method using simulated GW signals injected into Gaussian noise following the design sensitivities of the Advanced LIGO Advanced Virgo detectors. We compare the performance of this method with a traditionally-followed hierarchical inference approach. Both the methods are equally effective to hint at inhomogeneous populations, however, we find the mixture-likelihood approach to be more natural for mixture populations comprising compact objects of diverse classes. We also discuss the possible systematics in the mixture-likelihood approach, caused by several reasons, including the limited sensitivity of the second-generation detectors, specific features of the astrophysical population distributions, and the limitations posed by the waveform models employed. Finally, we apply this method to the LIGO-Virgo detections published in the second GW transient catalog (GWTC-2) and find them consistent with a binary black hole population with f(nbh) estimated to be <= 0.29 at 90% credibility.