# Cross-Validate a GAM

Cross-validation (CV) splits the data into folds, trains on all but one fold, and evaluates predictions on the held-out fold. Repeating this across all folds gives an honest estimate of how well the model will perform on new data.

Use CV when predictive accuracy on unseen data is the primary goal, or when you want a model-selection criterion that does not rely on asymptotic approximations. `wk.cross_validate()` handles the fold splitting and returns a summary score.


# Fit the candidate models

Load the `fish` dataset, which records fish counts by temperature and depth -- a natural Poisson response.


``` python
import whittaker as wk

# Load data and fit candidate models
data = wk.load_dataset("fish")

m_simple = wk.GAM("count ~ s(temperature)", family=wk.Poisson()).fit(data)
m_full   = wk.GAM("count ~ s(temperature) + s(depth)", family=wk.Poisson()).fit(data)
```


# Cross-validate the simple model

Run 5-fold CV on the simpler model. `cv_score` is the mean held-out deviance (lower is better); `cv_se` is its standard error across folds.


``` python
# Run 5-fold CV on the simple model
cv_simple = wk.cross_validate(
    "count ~ s(temperature)",
    data,
    family=wk.Poisson(),
    n_folds=5,
)
cv_simple.cv_score, cv_simple.cv_se
```


    (1.7706263709710162, 0.09754791292669326)


# Cross-validate the full model

Repeat for the model that also includes depth.


``` python
# Run 5-fold CV on the full model
cv_full = wk.cross_validate(
    "count ~ s(temperature) + s(depth)",
    data,
    family=wk.Poisson(),
    n_folds=5,
)
cv_full.cv_score, cv_full.cv_se
```


    (1.13030500590506, 0.08696847398477088)


# Compare the scores

A lower `cv_score` indicates better out-of-sample fit. However, because CV is a random estimate, a difference smaller than one standard error is not meaningful -- this is the "one-SE rule": the simplest model whose CV score falls within `best_score + best_se` is considered equivalent to the best model.


``` python
# Apply one-SE rule to compare models
threshold = cv_full.cv_score + cv_full.cv_se
cv_simple_within_1se = cv_simple.cv_score <= threshold

cv_full.cv_score, cv_simple.cv_score, threshold, cv_simple_within_1se
```


    (1.13030500590506, 1.7706263709710162, 1.2172734798898308, False)


If `cv_simple_within_1se` is `True`, the simpler model is not meaningfully worse and can be preferred for parsimony. If it is `False`, the extra term earns its place.


# Inspect the full model summary

When the fuller model wins, examine its summary to confirm both smooths are contributing.


``` python
m_full.summary()
```


    GAM fit summary
    ============================================================
    Formula:    count ~ s(temperature) + s(depth)
    Family:     Poisson(link='log')
    Inference:  GCV
    Observations: 300
    Coefficients: 19

    Parametric coefficients:
      Term                       Estimate    Std.Err    z value    p-value
      ------------------------ ---------- ---------- ---------- ----------
      (Intercept)                  1.5349     0.0289     53.128    < 1e-16

    Approximate significance of smooth terms:
      Term                        EDF Ref.df     Chi.sq    p-value
      ------------------------ ------ ------ ---------- ----------
      s(temperature)             4.24      5    294.642    < 1e-16
      s(depth)                   1.74      2    194.880    < 1e-16

    Total EDF:  6.98
    Scale est:  1.000000
    Deviance:   320.5707
    Null dev:   843.4578
    Dev. expl:  62.0%
    GCV score:  1.120093
    AIC:        1305.79
    BIC:        1331.65


Look at the EDF for each smooth. A term with EDF close to 1.0 is nearly linear; a term with very low EDF is barely contributing to predictions.


# CV vs AIC

| Criterion | Best for            | Notes                                     |
|-----------|---------------------|-------------------------------------------|
| AIC       | Model parsimony     | Fast; relies on asymptotic theory         |
| CV        | Predictive accuracy | Slower; honest estimate of generalisation |

Prefer CV when the deployment distribution matches your data and you care primarily about prediction. Prefer AIC when interpretability and parsimony matter most, or when data are too small for reliable fold estimation.