Conditional Matrix Flows for Gaussian Graphical Models

Studying conditional independence among many variables with few observations is a challenging task. Gaussian Graphical Models (GGMs) tackle this problem by encouraging sparsity in the precision matrix through l(q) regularization with q <= 1. However, most GMMs rely on the l(1) norm because the objective is highly non-convex for sub-l(1) pseudo-norms. In the frequentist formulation, the l(1) norm relaxation provides the solution path as a function of the shrinkage parameter lambda. In the Bayesian formulation, sparsity is instead encouraged through a Laplace prior, but posterior inference for different lambda requires repeated runs of expensive Gibbs samplers. Here we propose a general framework for variational inference with matrix-variate Normalizing Flow in GGMs, which unifies the benefits of frequentist and Bayesian frameworks. As a key improvement on previous work, we train with one flow a continuum of sparse regression models jointly for all regularization parameters lambda and all l(q) norms, including non-convex sub-l(1) pseudo-norms. Within one model we thus have access to (i) the evolution of the posterior for any lambda and any l(q) (pseudo-) norm, (ii) the marginal log-likelihood for model selection, and (iii) the frequentist solution paths through simulated annealing in the MAP limit.