Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

Bayesian methods

The solow-bayes crate fits Bayesian generalized linear mixed models (GLMMs) with a random-effects design. The single estimator type, BayesMixedGlm, supports two families — binomial (logit link) and Poisson (log link) — and offers two deterministic posterior approximations: mean-field variational Bayes via BayesMixedGlm::fit_vb (a factored-Gaussian posterior maximizing the evidence lower bound), and maximum a posteriori estimation via BayesMixedGlm::fit_map (the posterior mode). Both fits are fully deterministic — there is no Monte Carlo sampling.

Scope, honestly. This crate is narrower than the reference's full Bayesian surface: it covers the mixed-GLM variational/MAP estimators only. There is no MCMC sampler (no Gibbs or Metropolis), no posterior-draw machinery, and no credible-interval helper. Uncertainty is summarized by the variational posterior standard deviations returned alongside the means.

Background

For observation \(i\) with fixed-effects design row \(x_i\) and random-effects design row \(z_i\), the linear predictor is

\[ \eta_i = x_i^\top \beta + z_i^\top u, \]

where \(\beta\) are the fixed-effects coefficients (exog, length k_fep) and \(u\) are the random-effect realizations (exog_vc, length k_vc). The mean is \(\mu_i = g^{-1}(\eta_i)\): the logistic function for the binomial family and \(\exp(\cdot)\) for the Poisson family. The likelihood is the corresponding GLM density with its normalizing constant,

\[ \log p(y_i \mid \eta_i) = \begin{cases} y_i \eta_i - \log!\big(1 + e^{\eta_i}\big), & \text{binomial},\[4pt] y_i \eta_i - e^{\eta_i} - \log\Gamma(y_i + 1), & \text{Poisson}. \end{cases} \]

Each random effect \(u_j\) is a priori Gaussian with mean zero and standard deviation \(\exp(\theta_{,\mathrm{ident}[j]})\), where the variance-component parameters \(\theta\) (vcp, the log standard deviations) are shared across columns that map to the same ident value. The priors are

\[ u_j \sim \mathcal{N}!\big(0,\ e^{2\theta_{\mathrm{ident}[j]}}\big), \qquad \theta_k \sim \mathcal{N}(0,\ \texttt{vcp_p}^2), \qquad \beta_k \sim \mathcal{N}(0,\ \texttt{fe_p}^2). \]

Variational Bayes. fit_vb approximates the posterior over \((\beta, \theta, u)\) by a fully factored Gaussian \(q\) with independent coordinate means \(m\) and standard deviations \(s\), and maximizes the evidence lower bound

\[ \mathrm{ELBO}(q) = \mathbb{E}_q\big[\log p(y, \beta, \theta, u)\big] + \mathcal{H}(q). \]

The intractable expectation of the family term over the Gaussian linear predictor is evaluated by ten-point Gauss–Legendre quadrature. The analytic ELBO and its gradient are exposed as vb_elbo and vb_elbo_grad; the optimizer works on the reparameterized vector \([m;\ \log s]\) and minimizes \(-\mathrm{ELBO}\) by BFGS. Because the maximized ELBO is the model's variational log-likelihood, BayesMixedGlmResults::llf returns it as an alias for the elbo field.

MAP. fit_map instead locates the posterior mode — the \((\beta, \theta, u)\) maximizing the joint log-density \(\log p(y, \beta, \theta, u)\), exposed as log_posterior with gradient log_posterior_grad. This is a point estimate in the same dim()-dimensional space as a single variational-mean vector, with no posterior spread attached.

Example

A binomial random-intercept model: six groups of four observations each, one fixed-effects slope plus an intercept column, and a per-group random intercept (exog_vc is a group-indicator matrix, with a single shared variance component, so every ident entry is 0).

#![allow(unused)]
fn main() {
use ndarray::{Array1, Array2};
use solow_bayes::{BayesMixedGlm, BinomialBayesMixedGLM, Family};

let n_groups = 6usize;
let per = 4usize;
let n = n_groups * per;

// Fixed effects: intercept column + one centered covariate.
let mut exog = Array2::<f64>::zeros((n, 2));
// Random effects: one indicator column per group (the random intercepts).
let mut exog_vc = Array2::<f64>::zeros((n, n_groups));
for i in 0..n {
    exog[[i, 0]] = 1.0;
    exog[[i, 1]] = -1.5 + 3.0 * i as f64 / (n as f64 - 1.0);
    exog_vc[[i, i / per]] = 1.0;
}

let endog: Array1<f64> = ndarray::array![
    0., 0., 0., 1., 1., 0., 0., 0., 1., 0., 0., 1.,
    1., 1., 1., 1., 1., 0., 0., 1., 0., 0., 1., 1.
];

// All six random-intercept columns share one variance component.
let ident = vec![0usize; n_groups];

// Prior sds: vcp_p for the log-sd parameter, fe_p for the fixed effects.
let model = BinomialBayesMixedGLM::new(
    endog, exog, exog_vc, ident, /* vcp_p */ 0.5, /* fe_p */ 2.0,
)
.unwrap();

// Variational Bayes: default start (None, None), generous iteration budget.
let res = model.fit_vb(None, None, 100_000, 1e-10).unwrap();

println!("converged   = {}", res.converged);
println!("fe_mean     = {:?}", res.fe_mean);   // posterior means of [intercept, slope]
println!("fe_sd       = {:?}", res.fe_sd);     // posterior sds of the fixed effects
println!("vcp_mean    = {:?}", res.vcp_mean);  // log-sd of the random intercept
println!("vc_mean     = {:?}", res.vc_mean);   // per-group random-intercept means
println!("ELBO (llf)  = {:.4}", res.llf());
}

BinomialBayesMixedGLM::new is a thin named constructor that forwards to BayesMixedGlm::new(Family::Binomial, ..); use PoissonBayesMixedGLM::new (or pass Family::Poisson directly) for count responses. The returned BayesMixedGlmResults carries the posterior means fe_mean, vcp_mean, vc_mean and the matching posterior standard deviations fe_sd, vcp_sd, vc_sd, plus elbo, converged, iters, and grad_norm.

The same model can be fit at its posterior mode instead:

#![allow(unused)]
fn main() {
use ndarray::Array1;
use solow_bayes::BayesMixedGlm;
fn demo(model: &BayesMixedGlm) {
// MAP: maximize the joint log-density from the default deterministic start.
let map = model.fit_map(None, 5_000, 1e-10).unwrap();

println!("fe (mode)     = {:?}", map.fe);
println!("vcp (mode)    = {:?}", map.vcp);
println!("vc (mode)     = {:?}", map.vc);
println!("log-posterior = {:.4}", map.logposterior);

// The full stacked [fep, vcp, vc] vector is also available:
let stacked: &Array1<f64> = &map.params;
let _ = stacked;
}
}

Illustrative output: the binomial fit converges to a near-zero ELBO gradient and prints a two-element fe_mean (intercept and slope), a single vcp_mean log-standard-deviation for the shared random intercept, and six vc_mean per-group offsets; llf() reports the maximized ELBO. The exact printed numbers depend on the optimizer tolerance, so they are not reproduced here — run the example to see them.

Module reference

Models

NameDescription
BayesMixedGlmBayesian mixed GLM; construct with new, fit with fit_vb or fit_map.
BinomialBayesMixedGLMNamed constructor for a binomial (logit-link) mixed GLM.
PoissonBayesMixedGLMNamed constructor for a Poisson (log-link) mixed GLM.

Results

NameDescription
BayesMixedGlmResultsVariational posterior summary: fe_mean/vcp_mean/vc_mean, fe_sd/vcp_sd/vc_sd, elbo, converged, iters, grad_norm.
MapResultPosterior-mode summary: params, fe, vcp, vc, logposterior, converged, iters, grad_norm.

Functions (methods on BayesMixedGlm)

NameDescription
newBuild a model from family, endog, exog, exog_vc, ident, vcp_p, fe_p.
fit_vbMean-field variational Bayes fit; returns BayesMixedGlmResults.
fit_mapMaximum a posteriori (posterior-mode) fit; returns MapResult.
vb_elboEvidence lower bound at a (mean, sd) pair.
vb_elbo_gradGradient of the ELBO with respect to (mean, sd).
log_posteriorJoint log-density at a stacked [fep, vcp, vc] vector.
log_posterior_gradGradient of the joint log-density.
llf(On BayesMixedGlmResults) the maximized ELBO, as a variational log-likelihood.

Enums

NameDescription
FamilyThe GLM family: Family::Binomial (logit) or Family::Poisson (log).

Full API: see the generated rustdoc for solow-bayes.

References

  • A. Gelman, J. B. Carlin, H. S. Stern, D. B. Dunson, A. Vehtari, and D. B. Rubin. Bayesian Data Analysis, 3rd ed. Chapman & Hall/CRC, 2013.
  • D. M. Blei, A. Kucukelbir, and J. D. McAuliffe. "Variational Inference: A Review for Statisticians." Journal of the American Statistical Association, 112(518):859–877, 2017.
  • C. M. Bishop. Pattern Recognition and Machine Learning, ch. 10 (Approximate Inference). Springer, 2006.
  • C. E. McCulloch and S. R. Searle. Generalized, Linear, and Mixed Models, 2nd ed. Wiley, 2008.