Verification & methodology
Solow's central claim is not "these look like reasonable numbers" but "these are the same numbers an authoritative reference produces." This chapter explains how that claim is enforced, the naming discipline behind it, and how to add a new model with its own verification fixture.
The golden-fixture harness
Verification is built on golden fixtures: frozen numerical outputs captured
once from an authoritative Python reference and committed to the repository as
JSON. The Rust test suites read those committed files and assert that Solow
reproduces every value to a tight tolerance. Crucially, the tests need no
Python at test time — cargo test runs against the frozen numbers.
The flow has three pieces:
-
Generators —
tools/reference/gen_*.pyscripts call the reference (andnumpy/scipyfor the linear-algebra and distribution fixtures) and dump results totests/fixtures/*.json. There is roughly one generator per area (gen_models.py,gen_glm.py,gen_discrete.py,gen_duration.py,gen_multivariate.py, …), often with_extcompanions for extended coverage. -
Fixtures —
tests/fixtures/*.jsonhold the committed golden values: inputs, the reference's parameter estimates, standard errors, and the other reported statistics. -
Rust tests — each crate has a
tests/reference*.rsthat loads its fixture and compares Solow's output element-by-element.
tools/reference/gen_models.py ──► tests/fixtures/models.json ──► crates/solow-regression/tests/reference.rs
(run once, by a human) (committed JSON) (runs on every `cargo test`)
Tolerances
Agreement is asserted to a tolerance appropriate to each computation:
| Area | Tolerance |
|---|---|
| Most quantities (regression, distributions, tests) | 1e-8 |
| Maximum-likelihood / variational-Bayes estimates | 1e-6 |
| The formula engine (design matrices) | 1e-12 |
| The Python bindings vs. the Rust results | ~1e-12 |
The MLE tolerance is looser only because two independent optimizers converging
to the same optimum agree to optimizer precision, not to the last bit; the
underlying closed-form quantities still match to 1e-8 or better.
Regenerating fixtures
The numpy/scipy-based fixtures regenerate directly:
python3 tools/reference/gen_linalg.py
python3 tools/reference/gen_distributions.py
The modeling fixtures load the reference statistics package indirectly, by a module name supplied through an environment variable, so the repository never hard-codes that package's name:
SOLOW_REFERENCE=<reference-import-name> python3 tools/reference/gen_models.py
The naming discipline
This indirection is deliberate and is enforced as a project rule: the name of
the source statistics package never appears in the Solow source tree. The
reference is loaded through SOLOW_REFERENCE at fixture-generation time and is
referred to in prose only as "the reference" or "an authoritative reference."
Comparison against the broader scientific-Python ecosystem (numpy, scipy,
matplotlib, patsy) is named freely; those are general tools, not the
specific package Solow re-implements.
The reason is that Solow is an independent, from-scratch re-implementation, not a wrapper or a fork. Keeping the source name out of the tree keeps that boundary clean while still letting the test suite prove numerical equivalence.
Adding a model and its fixture
To add a new estimator with verification, follow the same pattern every existing crate uses:
-
Implement the model in its crate (e.g.
crates/solow-yourarea/src/), exposing a results struct with the standard fields (params,bse,tvalues,pvalues, information criteria, …). -
Write a generator
tools/reference/gen_yourarea.pythat fits the same model in the reference on a fixed input and writes the inputs plus the reference outputs totests/fixtures/yourarea.json. Load the reference via theSOLOW_REFERENCEenvironment variable — never import it by name. -
Generate the fixture once and commit the JSON:
SOLOW_REFERENCE=<reference-import-name> python3 tools/reference/gen_yourarea.py -
Add a Rust test
crates/solow-yourarea/tests/reference.rsthat reads the fixture, fits the model with Solow on the same input, and asserts each value agrees within the appropriate tolerance (use theapproxcrate'sassert_abs_diff_eq!). -
Run it offline:
cargo test -p solow-yourareanow verifies against the frozen numbers with no Python in the loop.
Because the fixtures are committed, contributors and CI verify correctness without any reference installation; only regenerating a fixture (when the input or the reference version changes) needs Python.
Verifying the snippets in this book
Every Rust snippet printed in this guide is itself compile-checked against the
real crate APIs. The examples are pasted into a scratch crate that depends on the
Solow crates by path and built with cargo build; a signature mismatch fails the
build. This keeps the documentation honest as the API evolves.