GAM that reads data from Polars LazyFrames, DataFrames, or files.
PolarsGAM(
formula,
family=None,
*,
n_discrete=200,
chunk_size=100000,
)
PolarsGAM extends ~whittaker.bam.BigGAM with a data-loading layer built on Polars, so it can source data from an in-memory Polars DataFrame/LazyFrame or directly from a file on disk (CSV, Parquet, IPC/Arrow, or NDJSON), without requiring the caller to materialize the whole dataset into a dict of NumPy arrays first. File paths are opened lazily (pl.scan_*), and the resulting LazyFrame is collected using Polars’ streaming query engine (collect(streaming=True)), which evaluates the query plan incrementally rather than loading the entire source at once. The collected frame is then walked in chunk_size-row slices (iter_slices) and each column is converted to a NumPy array and concatenated, producing the same dict[str, numpy.ndarray] that ~whittaker.gam.GAM.fit and ~whittaker.bam.BigGAM.fit expect. Fitting itself then proceeds exactly as in BigGAM: covariates are discretized to at most n_discrete unique values per smooth, and the design matrix is never materialized.
Use PolarsGAM when the data already lives in Polars, or on disk in a Polars-readable format, and you want to avoid a manual load-then-convert step — particularly for datasets in the 1M-100M row range where Polars’ streaming engine keeps peak memory bounded during the read. For SQL-native sources or datasets that are more naturally expressed as a database query (joins, filters, aggregations), see ~whittaker.duckdb.DuckDBGAM instead.
Requires the polars package (install via pip install whittaker[polars]).
Parameters
formula: str or Formula-
Model formula as a string (e.g. "y ~ s(x1) + s(x2) + x3"), or an already-parsed Formula object. Same syntax as ~whittaker.gam.GAM.
family: Family = None-
Response distribution family, e.g. Gaussian(), Binomial(), Poisson(), Gamma(), or Tweedie(). Defaults to Gaussian().
n_discrete: int = 200-
Maximum number of unique representative values per covariate used when discretizing smooth terms (see ~whittaker.bam.BigGAM). Defaults to 200.
chunk_size: int = 100000-
Number of rows collected per slice when converting the source into NumPy arrays. Smaller values reduce peak memory during the Polars-to-NumPy conversion step at the cost of more Python-level overhead; larger values reduce overhead but require more memory per slice. Defaults to
100_000.
Examples
import numpy as np
import polars as pl
from whittaker.polars_streaming import PolarsGAM
rng = np.random.default_rng(0)
n = 5_000
x1 = rng.uniform(0, 1, n)
x2 = rng.uniform(0, 1, n)
y = np.sin(2 * np.pi * x1) + x2**2 + rng.normal(scale=0.2, size=n)
df = pl.DataFrame({"x1": x1, "x2": x2, "y": y})
model = PolarsGAM("y ~ s(x1) + s(x2)", n_discrete=100, chunk_size=1_000)
model.fit(df.lazy())
print(model.summary())
GAM fit summary
============================================================
Formula: y ~ s(x1) + s(x2)
Family: Gaussian(link='identity')
Inference: REML
Observations: 0
Coefficients: 19
Parametric coefficients:
Term Estimate Std.Err t value p-value
------------------------ ---------- ---------- ---------- ----------
(Intercept) 0.3376 0.0000 0.000 1
Approximate significance of smooth terms:
Term EDF Ref.df Chi.sq p-value
------------------------ ------ ------ ---------- ----------
s(x1) 8.91 9 62254.971 < 1e-16
s(x2) 6.26 7 11059.123 < 1e-16
Total EDF: 16.17
Scale est: 0.040068
Deviance: 199.6941
Null dev: 3148.4427
Dev. expl: 93.7%
GCV score: 0.040198
AIC: -1880.28
BIC: -1774.91
A file path can be passed directly instead of a DataFrame/LazyFrame — PolarsGAM infers the format from the extension and scans it lazily:
model = PolarsGAM("y ~ s(x)")
model.fit("large_dataset.parquet")
Fitting from an actual multi-gigabyte file requires pip install whittaker[polars] and enough disk I/O bandwidth to stream the file; the in-memory example above is kept small so it runs quickly, but the same code path scales to files with tens of millions of rows.
Attributes
|
Name
|
Description
|
|
chunk_size
|
Chunk size used when converting the Polars source to NumPy arrays.
|
|
n_rows
|
Total number of rows in the source used by the most recent fit.
|
chunk_size
Chunk size used when converting the Polars source to NumPy arrays.
This is the chunk_size value passed to __init__: the number of rows per slice that _lazyframe_to_dict requests from LazyFrame.iter_slices while converting the collected frame into the dict[str, numpy.ndarray] consumed by fitting.
n_rows
Total number of rows in the source used by the most recent fit.
Populated by fit() via _count_lazy, which evaluates lf.select(pl.len()) with Polars’ streaming engine before collecting the data. Remains 0 until fit() has been called at least once.
Methods
|
Name
|
Description
|
|
fit()
|
Fit the GAM from a Polars source.
|
fit()
Fit the GAM from a Polars source.
fit(
data,
*,
smoothing_params=None,
method="fREML",
select=False,
**kwargs,
)
Converts data to a Polars LazyFrame, collects it via Polars’ streaming engine in chunk_size-row slices, and combines the result into dict[str, numpy.ndarray] before delegating to ~whittaker.bam.BigGAM’s discretized fitting (see the class docstring for details on the discretization and cross-product accumulation).
Parameters
data: polars.LazyFrame or polars.DataFrame or str or pathlib.Path-
The data source. A Polars LazyFrame is used as-is; a DataFrame is converted with .lazy(); a file path (string or Path) is scanned lazily based on its extension — .parquet (scan_parquet), .csv (scan_csv), .ipc/.arrow (scan_ipc), or .ndjson/.jsonl (scan_ndjson). Any other type or unrecognized extension raises TypeError or ValueError respectively.
smoothing_params: list of float = None-
Fixed smoothing parameters lambda_j, one per smooth term, in formula order. If None (the default), smoothing parameters are selected automatically according to method.
method: str = "fREML"-
Criterion used to select smoothing parameters when smoothing_params is None. One of "fREML" (default), "REML", "ML", or "GCV". See ~whittaker.gam.GAM.fit for a description of each criterion.
select: bool = False-
If True, add an extra penalty on each smooth’s null space so the term can be shrunk to exactly zero (double-penalty selection). Defaults to False.
**kwargs-
Reserved for future keyword arguments; currently unused.
Returns
PolarsGAM-
Returns
self for method chaining, e.g. model = PolarsGAM("y ~ s(x)").fit(data).