Choose a Smoothing Method

Compare REML, GCV, and ML and know when each is appropriate.

The smoothing method controls how the penalty strength λ is selected during fitting. A well-chosen λ balances the fidelity of the smooth to the data against its wiggliness. Whittaker offers three methods: REML (the default), GCV, and ML. All three produce a fitted model but they differ in their statistical criterion and their tendency to over or undersmooth.

Fit with each method

The method argument is passed to fit(). The dataset is mcycle, which deals with head acceleration over time during a simulated crash.

import whittaker as wk

# Load data and fit with each smoothing method
data = wk.load_dataset("mcycle")

model_reml = wk.GAM("accel ~ s(times)").fit(data, method="REML")
model_gcv  = wk.GAM("accel ~ s(times)").fit(data, method="GCV")
model_ml   = wk.GAM("accel ~ s(times)").fit(data, method="ML")

Smoothing parameters

Each fitted model exposes smoothing_params: the selected λ values (one per smooth). A larger λ compresses the smooth toward a straight line; a smaller λ allows more wiggle. Compare the value selected by each method.

# REML
model_reml.smoothing_params[0]

266.83963427768026

# GCV
model_gcv.smoothing_params[0]

371.6516581290851

# ML
model_ml.smoothing_params[0]

266.83963427768026

GCV often selects a smaller λ than REML, meaning it allows a wigglier smooth. REML and ML tend to be closer to each other.

Effective degrees of freedom

edf measures the wiggliness of each smooth: higher EDF means more curvature. Lower means more compression toward a line. Check how much each method differs.

# REML
model_reml.edf[0]

8.151247447768561

# GCV
model_gcv.edf[0]

7.915324259501651

# ML
model_ml.edf[0]

8.151247447768561

Small differences in EDF across methods are normal. Large differences (more than a unit or two) suggest the criterion matters for this dataset and REML is usually the safer choice.

Full summaries

Read the full summaries to compare deviance explained, GCV score, and AIC across methods.

model_reml.summary()

GAM fit summary
============================================================
Formula:    accel ~ s(times)
Family:     Gaussian(link='identity')
Inference:  REML
Observations: 133
Coefficients: 10

Parametric coefficients:
  Term                       Estimate    Std.Err    t value    p-value
  ------------------------ ---------- ---------- ---------- ----------
  (Intercept)                -45.6924     1.8352    -24.898    < 1e-16

Approximate significance of smooth terms:
  Term                        EDF Ref.df     Chi.sq    p-value
  ------------------------ ------ ------ ---------- ----------
  s(times)                   8.15      9    667.987    < 1e-16

Total EDF:  9.15
Scale est:  447.915129
Deviance:   55473.7300
Null dev:   357878.4929
Dev. expl:  84.5%
GCV score:  481.011806
AIC:        1198.50
BIC:        1224.95

model_gcv.summary()

GAM fit summary
============================================================
Formula:    accel ~ s(times)
Family:     Gaussian(link='identity')
Inference:  GCV
Observations: 133
Coefficients: 10

Parametric coefficients:
  Term                       Estimate    Std.Err    t value    p-value
  ------------------------ ---------- ---------- ---------- ----------
  (Intercept)                -45.6924     1.8364    -24.882    < 1e-16

Approximate significance of smooth terms:
  Term                        EDF Ref.df     Chi.sq    p-value
  ------------------------ ------ ------ ---------- ----------
  s(times)                   7.92      8    663.379    < 1e-16

Total EDF:  8.92
Scale est:  448.520495
Deviance:   55654.5201
Null dev:   357878.4929
Dev. expl:  84.4%
GCV score:  480.746115
AIC:        1198.44
BIC:        1224.21

model_ml.summary()

GAM fit summary
============================================================
Formula:    accel ~ s(times)
Family:     Gaussian(link='identity')
Inference:  ML
Observations: 133
Coefficients: 10

Parametric coefficients:
  Term                       Estimate    Std.Err    t value    p-value
  ------------------------ ---------- ---------- ---------- ----------
  (Intercept)                -45.6924     1.8352    -24.898    < 1e-16

Approximate significance of smooth terms:
  Term                        EDF Ref.df     Chi.sq    p-value
  ------------------------ ------ ------ ---------- ----------
  s(times)                   8.15      9    667.987    < 1e-16

Total EDF:  9.15
Scale est:  447.915129
Deviance:   55473.7300
Null dev:   357878.4929
Dev. expl:  84.5%
GCV score:  481.011806
AIC:        1198.50
BIC:        1224.95

Comparison table

Method	Criterion	Tendency	When to use
REML	Restricted maximum likelihood	Balanced; slightly conservative	Default, most datasets
GCV	Generalized cross-validation	Can undersmooth (too wiggly)	Quick exploration, large n
ML	Maximum likelihood	Similar to REML, no correction for fixed effects	Comparing models with different fixed-effect structures

Rule of thumb

It’s often good to use REML unless you have a specific reason to switch to something else. GCV is a reasonable alternative for large datasets where speed matters and mild undersmoothing is acceptable. Use ML only when you need to compare models that differ in their parametric (non-smooth) terms. ML (unlike REML) produces a likelihood that is directly comparable across such models.