Single Deterministic Neural Network with Hierarchical Gaussian Mixture Model for Uncertainty Quantification

We propose a method to train a deterministic deep network for uncertainty quantification (UQ) with a single forward pass. Traditional Monte Carlo or ensemble based UQ methods largely leverage the variation of neural network weights to introduce uncertainty. We propose a hierarchical Gaussian mixture model (GMM) based nonlinear classifier to shape the extracted feature more flexibly and express the uncertainty by the entropy of the predicted posterior distribution. We perform large-scale training with this hierarchical GMM based loss function and introduce a natural gradient descent algorithm to update the parameters of the hierarchical GMM. With a single deterministic neural network, our uncertainty quantification approach performs well when training and testing on large datasets. We show competitive performance scores on several benchmark datasets and the out-of-distribution detection task on notable challenging dataset pairs such as CIFAR-10 vs. STL10/SVHN, and CIFAR100 vs. STL10/SVHN.