import numpy as np
import whittaker as wk
# Simulate large dataset with two covariates
rng = np.random.default_rng(23)
n = 500_000
x1 = rng.uniform(0, 2 * np.pi, n)
x2 = rng.uniform(0, 1, n)
y = np.sin(x1) + 0.5 * x2 + rng.normal(0, 0.3, n)
data = {"y": y, "x1": x1, "x2": x2}Fit a BigGAM on a Large DataFrame
Use BigGAM to fit a smooth model on millions of rows without materializing the full design matrix.
Standard GAMs build a design matrix with one row per observation, which becomes prohibitively large for datasets with millions of rows. BigGAM sidesteps this by discretizing each covariate into bins before fitting, reducing memory from O(n·p) to O(d·p) where d is the number of bins (typically 200 or fewer).
Generate Data
Simulate a large dataset with two covariates and a response that combines a sinusoidal and a linear effect.
Fit
BigGAM accepts the same formula syntax as a regular GAM. The default method is "fREML", which is better suited to discretized data than "REML".
model = wk.BigGAM("y ~ s(x1, k=10) + s(x2, k=10)").fit(data)
model.summary()GAM fit summary
============================================================
Formula: y ~ s(x1, k=10) + s(x2, k=10)
Family: Gaussian(link='identity')
Inference: REML
Observations: 0
Coefficients: 19
Parametric coefficients:
Term Estimate Std.Err t value p-value
------------------------ ---------- ---------- ---------- ----------
(Intercept) 0.2503 0.0000 0.000 1
Approximate significance of smooth terms:
Term EDF Ref.df Chi.sq p-value
------------------------ ------ ------ ---------- ----------
s(x1, k=10) 9.00 9 2783105.298 < 1e-16
s(x2, k=10) 1.02 2 115841.852 < 1e-16
Total EDF: 11.01
Scale est: 0.089773
Deviance: 44885.4985
Null dev: 305380.4545
Dev. expl: 85.3%
GCV score: 0.089775
AIC: 213713.90
BIC: 213836.39Predict
Prediction works identically to a standard GAM: pass a dict of arrays for the covariates of interest.
new_data = {"x1": np.array([0.5, 1.0, 2.0]), "x2": np.array([0.2, 0.5, 0.8])}
preds = model.predict(new_data)
preds.valuesarray([0.57799383, 1.09238007, 1.30912694])Interpret
BigGAM discretizes each covariate into at most n_discrete=200 unique bins, reducing the fitting problem from O(n·p) to O(d·p) where d ≪ n. The smooth estimates are essentially identical to a full GAM fitted on the raw data, because the precision gained from additional observations beyond the bin resolution is negligible. This makes BigGAM a practical drop-in for GAM when data is too large to hold a full design matrix in memory (typically when n exceeds a few hundred thousand rows).