Compare Models with AIC

Use AIC to choose between nested GAMs of varying complexity.

The AIC (Akaike Information Criterion) ranks models by balancing goodness-of-fit against complexity. In a GAM, complexity is measured by the total effective degrees of freedom (EDF) rather than the raw number of parameters, so AIC automatically accounts for how wiggly each smooth is.

Use AIC when you have several candidate models and want a principled way to decide which predictors to include.

Fit the candidate models

Load the abalone dataset and fit three nested models, starting with a single smooth and adding predictors one at a time.

import whittaker as wk

# Load data and fit nested candidate models
data = wk.load_dataset("abalone")

m1 = wk.GAM("rings ~ s(length)").fit(data)
m2 = wk.GAM("rings ~ s(length) + s(diameter)").fit(data)
m3 = wk.GAM("rings ~ s(length) + s(diameter) + s(height) + s(shucked_weight)").fit(data)

Compare AIC values

Extract the .aic attribute from each model and display them together.

# Compare AIC values across models
m1.aic, m2.aic, m3.aic

(2052.895482172216, 2054.8931824650294, 1847.3973234458235)

Lower AIC is better. A difference (ΔAIC) of less than 2 means the models are essentially equivalent in predictive quality. A ΔAIC greater than 10 means the simpler model is decisively worse and the extra predictors are earning their place.

Check total EDF

AIC penalises via total EDF, so it is worth inspecting how complex each model actually is.

# Compare total EDF across models
m1.edf_total, m2.edf_total, m3.edf_total

(4.329619009488265, 5.339006719376878, 19.625034023054337)

The full model carries more EDF. If its AIC is not substantially lower, the extra terms are not paying off after the penalty is applied.

Inspect the winning model

Look at the summary of the winning model. In this dataset m3 has the lowest AIC.

m3.summary()

GAM fit summary
============================================================
Formula:    rings ~ s(length) + s(diameter) + s(height) + s(shucked_weight)
Family:     Gaussian(link='identity')
Observations: 500
Coefficients: 37

Parametric coefficients:
  Term                       Estimate    Std.Err    t value    p-value
  ------------------------ ---------- ---------- ---------- ----------
  (Intercept)                  6.8680     0.0673    102.038    < 1e-16

Approximate significance of smooth terms:
  Term                        EDF Ref.df     Chi.sq    p-value
  ------------------------ ------ ------ ---------- ----------
  s(length)                  6.27      7     42.371  4.411e-07
  s(diameter)                1.00      2      0.002     0.9991
  s(height)                  5.99      6     25.650  0.0002587
  s(shucked_weight)          5.36      6    245.380    < 1e-16

Total EDF:  19.63
Deviance:   1088.1502
Null dev:   6745.2880
Dev. expl:  83.9%
GCV score:  2.357752
Scale est:  2.265210
AIC:        1847.40
BIC:        1930.11

Check deviance_explained and the per-smooth EDF values. Any smooth with EDF near 1.0 is nearly linear (it contributes little beyond a simple slope).

AIC rule of thumb

ΔAIC from best model	Interpretation
0–2	Essentially equivalent; prefer simpler
2–10	Some evidence for the more complex model
> 10	Strong evidence; simpler model is decisively worse

AIC selects the model that will generalise best on average, not the model that fits the training data best. When predictive accuracy on held-out data is the primary goal, pair AIC comparison with cross-validation (see recipe 26).