Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

Model evaluation metrics

solow-metrics bundles every number you report after an estimator has been fit — losses and scores for regression, calibration diagnostics for probabilistic classifiers, forecast-comparison tests, non-parametric model comparison, Bayesian model comparison, and model-agnostic interpretability tools. Every metric agrees with its canonical definition (or the published reference paper) to machine precision on the fixture suite in tests/fixtures/metrics/*.json.

The functions all take ArrayView inputs so they compose with slices, subranges, and matrix rows without copies. Every metric returns a Result — mismatched shapes, negative sample weights, unknown classes, or a constant target for R² are user errors, not silent NaNs.

Regression

MetricCall
Mean squared errormean_squared_error(y, ŷ, w?)
Root mean squared errorroot_mean_squared_error(y, ŷ, w?)
Mean absolute errormean_absolute_error(y, ŷ, w?)
Median absolute errormedian_absolute_error(y, ŷ)
Max absolute residualmax_error(y, ŷ)
MAPE / sMAPEmean_absolute_percentage_error(y, ŷ, w?), symmetric_mean_absolute_percentage_error(y, ŷ, w?)
Squared / root squared log errormean_squared_log_error(y, ŷ, w?), root_mean_squared_log_error(y, ŷ, w?)
R²r2_score(y, ŷ, w?)
Explained varianceexplained_variance_score(y, ŷ, w?)
Mean pinball loss (quantile)mean_pinball_loss(y, ŷ, α, w?)
D² absolute / D² Tweedied2_absolute_error_score(y, ŷ, w?), d2_tweedie_score(y, ŷ, w?, power)
Huber losshuber_loss(y, ŷ, δ, w?)
Log-cosh losslog_cosh_loss(y, ŷ, w?)
One-call bundleRegressionReport::compute(y, ŷ, w?)

Core sums use the Kahan/Neumaier compensated accumulator from solow-core::numeric, so long residual sums of mixed magnitudes do not lose low-order bits.

GLM / Tweedie deviance

MetricCall
Tweedie deviance (any power)mean_tweedie_deviance(y, ŷ, w?, power)
Poisson deviancemean_poisson_deviance(y, ŷ, w?)
Gamma deviancemean_gamma_deviance(y, ŷ, w?)
D² (deviance-based)d2_tweedie_score(y, ŷ, w?, power)

These are the natural goodness-of-fit numbers to pair with a solow-glm::Glm fitted on Poisson or Gamma data.

Classification

Labels are usize class indices in [0, k). Probabilities live in a (n, k) matrix that must sum to one row-wise for probability metrics.

GroupCalls
Confusion matrixconfusion_matrix(y, ŷ, num_classes?)
Accuracy familyaccuracy_score, zero_one_loss, balanced_accuracy_score
Precision / Recall / Fβprecision_recall_fscore(y, ŷ, Average, β, w?), fbeta_score(...)
AveragingAverage::Binary / Macro / Weighted / Micro
Multi-class agreementmatthews_corrcoef, cohen_kappa_score(y, ŷ, KappaWeights)
Log-loss familybinary_log_loss(y, p, w?, ε), log_loss(y, p, w?, ε)
Focal lossesbinary_focal_loss(y, p, γ, α, w?), multiclass_focal_loss(y, p, γ, α, w?)
Score / marginhinge_loss(y, decision, w?), brier_score_loss(y, p, w?)
ROC / PRroc_curve(y, score), roc_auc_score(y, score), precision_recall_curve(y, score), average_precision_score(y, score)
Multiclass ROC-AUCroc_auc_ovr_score(y, p, MulticlassAuc::{Macro,Weighted}), roc_auc_ovo_score(...) (Hand-Till)
Top-ktop_k_accuracy_score(y, p, k)
Multiclass Brier / RPSmulticlass_brier_score(y, p), ranked_probability_score(y, p)
Top-1 ECEtop_label_calibration_error(y, p, n_bins)

Calibration diagnostics and post-hoc calibrators

Diagnostics — how well a probability means what it says:

DiagnosticCall
Reliability curvereliability_curve(y, p, n_bins, BinStrategy::{Uniform,Quantile})
Expected / Maximum CEexpected_calibration_error(...), maximum_calibration_error(...)
Bröcker decompositionbrier_decomposition(y, p, n_bins, strategy) — reliability + resolution + uncertainty + within-bin variance + raw Brier

Calibrators — train on held-out scores and re-map:

CalibratorFitTransform
Platt (1999)PlattScaling::fit(s, y).transform(s)
Isotonic (PAV)IsotonicRegression::fit(s, y).transform(s)
Temperature (GPSW 2017)TemperatureScaling::fit(logits, y).transform(logits)

Forecasting

MetricCall
MASEmase(y_true, y_pred, y_train, m)
RMSSErmsse(y_true, y_pred, y_train, m)
Pinball losspinball_loss(y, ŷ, α)
Interval coverageinterval_coverage(y, lo, hi)
Mean Winkler interval scoremean_interval_score(y, lo, hi, α)
Diebold-Mariano (HLN-corrected)diebold_mariano(y, f1, f2, h, DmLoss)
Giacomini-White conditionalgiacomini_white_test(y, f1, f2, test_regressors, h, DmLoss)

Distribution-free prediction intervals (conformal)

Both give finite-sample coverage guarantees under exchangeability, no distributional assumption required.

MethodFitInterval
Split conformalSplitConformal::fit(residuals, α).interval(ŷ) → PredictionInterval
Jackknife+ (BCRT 2021)JackknifePlus::new(loo_residuals, α).interval(loo_predictions)

Model comparison

Non-parametric — the Demšar (2006) recipe on an (m × k) score matrix (datasets × models, higher-is-better):

  • friedman_test(scores) → Iman-Davenport F-adjusted χ² with mean ranks and an F(k−1, (k−1)(m−1)) p-value.
  • nemenyi_critical_difference(k, m, α) → tabulated critical distance at α ∈ {0.05, 0.10}.
  • wilcoxon_signed_rank(a, b) → paired, tie-corrected normal approximation, scipy-compatible.

Bayesian — from a (n_samples × n_observations) posterior log-likelihood matrix:

  • waic(log_lik) → WaicResult { elpd, elpd_se, p_waic, waic, pointwise }.
  • psis_loo(log_lik) → PsisLooResult { elpd, elpd_se, p_loo, looic, pointwise, pareto_k }. The per-observation Pareto tail-shape k̂ is the standard reliability flag — anything > 0.7 marks an LOO estimate that should be refit.

Model-agnostic interpretability

DiagnosticCallReturns
Permutation importancepermutation_importance(x, scorer, n_repeats, seed)Vec<FeatureImportance>
Partial dependencepartial_dependence(x, feature, grid, predictor)PartialDependence
Accumulated local effectsaccumulated_local_effects(x, feature, n_bins, predictor)AccumulatedLocalEffects

The estimator is decoupled: pass any Fn(ArrayView2) -> Result<f64> scorer / Fn(ArrayView2) -> Result<Array1<f64>> predictor and this layer never sees the underlying model.

Serialization

Every public result / report struct derives serde::Serialize / serde::Deserialize behind the opt-in serde feature:

solow-metrics = { version = "0.2", features = ["serde"] }

A full evaluation loop can then be persisted as JSON and diffed across runs, model versions, or CI jobs.

Reference-fixture pipeline

tests/fixtures/metrics/*.json holds committed golden reference outputs. The Rust replay tests re-verify each fixture on every CI run and fail on drift, guaranteeing the shipped metrics stay bit-wise identical to their canonical mathematical definition.

Module reference