Update a Streaming GAM Incrementally

Fit a GAM in an online fashion by calling partial_fit() on successive data batches, then solve() to update the model.

StreamingGAM is designed for data that arrives in chunks, like sensor feeds, log streams, or any dataset too large to hold in memory at once. Each call to partial_fit() accumulates sufficient statistics from one batch, and solve() computes the current best-fit model without revisiting earlier batches.

Fit

Create the model once, then loop over incoming batches. Calling solve() after each partial_fit() is optional but lets you inspect the model’s state as data accumulates.

import numpy as np
import whittaker as wk

rng = np.random.default_rng(23)
n_per_batch = 200

# Initialize streaming model
model = wk.StreamingGAM("y ~ s(x, k=10)")

# Feed batches and solve incrementally
for i in range(5):
    x = rng.uniform(0, 2 * np.pi, n_per_batch)
    y = np.sin(x) + rng.normal(0, 0.3, n_per_batch)
    model.partial_fit({"x": x, "y": y})
    model.solve()
    print(f"Batch {i+1}: n_obs={model.n_obs}, edf={model.edf_total:.2f}")

Batch 1: n_obs=200, edf=8.30
Batch 2: n_obs=400, edf=8.92
Batch 3: n_obs=600, edf=9.22
Batch 4: n_obs=800, edf=9.38
Batch 5: n_obs=1000, edf=9.48

Summarize

After all batches have been processed, inspect the final model. The summary reflects the full accumulated dataset.

model.summary()

'StreamingGAM summary\n============================================================\nFormula:     y ~ s(x, k=10)\nFamily:      Gaussian\nDecay:       1.0\nN obs:       1000\nN batches:   5\nEDF total:   9.5\nScale:       0.3639\nDeviance:    360.5\n\nSmooth terms:\n  s(x, k=10): edf=8.5\n\nSnapshots:   5'

Predict

With the model solved, generate predictions on new inputs just as you would with a standard GAM.

new_data = {"x": np.linspace(0, 2 * np.pi, 100)}
preds = model.predict(new_data)
preds.values[:5]

array([-0.12063   , -0.04200138,  0.03661861,  0.11504262,  0.19281312])

Interpret

Incremental fitting is useful when data arrives in a stream or is too large to hold in memory. Because StreamingGAM accumulates sufficient statistics rather than raw observations, memory usage stays constant regardless of how many batches have been processed. Setting decay < 1.0 applies exponential downweighting to older batches, making the model responsive to distributional shifts over time.