Stream a Polars LazyFrame

Fit a GAM on a Polars LazyFrame, deferring data loading until fit time.

PolarsGAM integrates with Polars’ lazy execution model, and you build up a query plan as a LazyFrame, then pass it directly to fit(). Data is materialized and streamed in chunks only when fitting begins, so complex transformations upstream are folded into a single efficient scan.

Prepare a LazyFrame

Build a large synthetic dataset as a Polars DataFrame, then convert it to a LazyFrame. In a real workflow this might point at a Parquet file or a query chain that has not yet executed.

import numpy as np
import polars as pl
import whittaker as wk

# Build synthetic Polars DataFrame
rng = np.random.default_rng(23)
n = 200_000
df = pl.DataFrame({
    "x1": rng.uniform(0, 2 * np.pi, n),
    "x2": rng.uniform(0, 1, n),
    "y": np.sin(rng.uniform(0, 2 * np.pi, n)) + 0.5 * rng.uniform(0, 1, n) + rng.normal(0, 0.3, n),
})

# Convert to LazyFrame for deferred execution
lazy = df.lazy()

Fit

Pass the LazyFrame directly to fit(). PolarsGAM triggers execution internally and streams it through the discretized-basis fitting algorithm.

model = wk.PolarsGAM("y ~ s(x1, k=10) + s(x2, k=8)").fit(lazy)
model.summary()

GAM fit summary
============================================================
Formula:    y ~ s(x1, k=10) + s(x2, k=8)
Family:     Gaussian(link='identity')
Inference:  REML
Observations: 0
Coefficients: 17

Parametric coefficients:
  Term                       Estimate    Std.Err    t value    p-value
  ------------------------ ---------- ---------- ---------- ----------
  (Intercept)                  0.2523     0.0000      0.000          1

Approximate significance of smooth terms:
  Term                        EDF Ref.df     Chi.sq    p-value
  ------------------------ ------ ------ ---------- ----------
  s(x1, k=10)                1.00      2      0.258      0.879
  s(x2, k=8)                 1.01      2      0.189     0.9099

Total EDF:  3.01
Scale est:  0.613000
Deviance:   122598.2194
Null dev:   122598.4989
Dev. expl:  0.0%
GCV score:  0.613010
AIC:        469700.47
BIC:        469731.23

Inspect Row Count

After fitting, n_rows reports how many observations were processed.

model.n_rows

Predict

Predictions use a plain dict of arrays, with no Polars dependency at inference time.

new_data = {"x1": np.array([0.5, 1.5, 2.5]), "x2": np.array([0.2, 0.5, 0.8])}
preds = model.predict(new_data)
preds.values

array([0.25175845, 0.25145741, 0.25118906])

Interpret

PolarsGAM uses the same discretized-basis approach as BigGAM, so memory usage is bounded regardless of n. Passing a LazyFrame defers Polars query execution until fit() triggers it, meaning any upstream filter, join, or derived-column step runs only once as part of the streaming scan. PolarsGAM also accepts file paths directly (.parquet, .csv, .ipc, .arrow, .ndjson, and .jsonl extensions are all supported).