Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

Weighted & generalized least squares (WLS / GLS)

When the error variance is not constant across observations (heteroskedasticity), ordinary least squares stays unbiased but is no longer efficient, and its standard errors are wrong. Weighted least squares fixes this by weighting each observation by the inverse of its error variance. Generalized least squares is the general form: supply the full error covariance matrix Σ. With a diagonal Σ, GLS reduces exactly to WLS.

This example builds data whose noise standard deviation grows with x, then fits OLS, WLS, and GLS for comparison.

Code

#![allow(unused)]
fn main() {
use ndarray::{Array1, Array2};
use solow_core::tools::{add_constant, HasConstant};
use solow_regression::LinearModel;

// Heteroskedastic data: the noise sd scales with x, so later points are noisier.
let n = 60usize;
let x_raw: Vec<f64> = (0..n).map(|i| 1.0 + i as f64 * 0.5).collect();
let sd: Vec<f64> = x_raw.iter().map(|&xi| 0.35 * xi).collect();
// y_vec = 1 + 0.8 x + sd * noise

let x = Array2::from_shape_vec((n, 1), x_raw.clone()).unwrap();
let y = Array1::from(y_vec.clone());
let design = add_constant(&x, true, HasConstant::Add).unwrap();

// Weights proportional to 1 / variance; Sigma is the diagonal variance matrix.
let weights = Array1::from(sd.iter().map(|&s| 1.0 / (s * s)).collect::<Vec<_>>());
let mut sigma = Array2::<f64>::zeros((n, n));
for i in 0..n { sigma[[i, i]] = sd[i] * sd[i]; }

let ols = LinearModel::ols(y.clone(), design.clone()).unwrap().fit().unwrap();
let wls = LinearModel::wls(y.clone(), design.clone(), weights).unwrap().fit().unwrap();
let gls = LinearModel::gls(y.clone(), design.clone(), &sigma).unwrap().fit().unwrap();
}

Printed summary

OLS treats every point as equally informative:

                            OLS Regression Results
==============================================================================
Dep. Variable:                       y   R-squared:                     0.565
Model:                             OLS   Adj. R-squared:                0.558
No. Observations:                    60   F-statistic:                   75.47
==============================================================================
                   coef    std err         t     P>|t|      [0.025      0.975]
------------------------------------------------------------------------------
const            0.7950      1.555     0.511     0.611      -2.317       3.907
x                0.7514      0.086     8.687     0.000       0.578       0.925
==============================================================================

WLS downweights the noisy high-x points, tightening the standard errors:

                            WLS Regression Results
==============================================================================
Dep. Variable:                       y   R-squared:                     0.743
Model:                             WLS   Adj. R-squared:                0.739
No. Observations:                    60   F-statistic:                   167.8
==============================================================================
                   coef    std err         t     P>|t|      [0.025      0.975]
------------------------------------------------------------------------------
const            1.0254      0.277     3.703     0.000       0.471       1.580
x                0.7345      0.057    12.955     0.000       0.621       0.848
==============================================================================

The WLS intercept standard error drops from 1.555 to 0.277. And GLS with a diagonal Σ reproduces the WLS coefficients to the bit:

GLS with diagonal Sigma reproduces the WLS coefficients exactly:
  WLS params = [1.025374, 0.734467]
  GLS params = [1.025374, 0.734467]

Plot

The dashed OLS line is tilted by the high-variance points; the solid WLS line tracks the underlying relationship more faithfully.

WLS vs OLS under heteroskedasticity