Mixture modeling within the context of pharmacokinetic (PK)/pharmacodynamic (PD) mixed effects modeling is a useful tool to explore a population for the presence of two or more subpopulations, not explained by evaluated covariates. At present, statistical tests for the existence of mixed populations have not been developed. Therefore, a simulation study was undertaken to evaluate mixture modeling with NONMEM and explore the following questions. First, what is the probability of concluding that a mixed population exists when there truly is not a mixture (false positive significance level)? Second, what is the probability of concluding that a mixed population (two subpopulations) exists when there is truly a mixed population (power), and how well can the mixture be estimated, both in terms of the population parameters and the individual subject classification. Seizure count data were simulated using a Poisson distribution such that each subject's count could decrease from its baseline value, as a function of dose via an Emax model. The dosing design for the simulation was based on a trial with the investigational anti-epileptic drug pregabalin. Four hundred and forty seven subjects received pregabalin as add on therapy for partial seizures, each with a baseline seizure count and up to three subsequent seizure counts. For the mixtures, the two subpopulations were simulated to differ in their Emax values and relative proportions. One subpopulation always had its Emax set to unity (Emax hi), allowing the count to approach zero with increasing dose. The other subpopulation was allowed to vary in its Emax value (Emax lo=0.75, 0.5, 0.25, and 0) and in its relative proportion (pr) of the population (pr=0.05, 0.10, 0.25, and 0.50) giving a total of 4 ⋅ 4=16 different mixtures explored. Three hundred data sets were simulated for each scenario and estimations performed using NONMEM. Metrics used information about the parameter estimates, their standard errors (SE), the difference between minimum objective function (MOF) values for mixture and non-mixture models (MOF(δ)), the proportion of subjects classified correctly, and the estimated conditional probabilities of a subject being simulated as having Emax lo (Emax hi) given that they were estimated as having Emaxlo (Emax hi) and being estimated as having Emaxlo (Emax hi) given that they were simulated as having Emax lo (Emax hi). The false positive significance level was approximately 0.04 (using all 300 runs) or 0.078 (using only those runs with a successful covariance step), when there was no mixture. When simulating mixed data and for those characterizations with successful estimation and covariance steps, the median (range) percentage of 95% confidence intervals containing the true values for the parameters defining the mixture were 94% (89–96%), 89.5% (58–96%), and 95% (92–97%) for pr, Emax lo, and Emax hi, respectively. The median value of the estimated parameters pr, Emax lo (excluding the case when Emax lo was simulated to equal 0) and Emax hi within a scenario were within ±28% of the true values. The median proportion of subjects classified correctly ranged from 0.59 to 0.96. In conclusion, when no mixture was present the false positive probability was less than 0.078 and when mixtures were present they were characterized with varying degrees of success, depending on the nature of the mixture. When the difference between subpopulations was greater (as Emax lo approached zero or pr approached 0.5) the mixtures became easier to characterize.
