# Fit a BigGAM on a Large DataFrame

Standard GAMs build a design matrix with one row per observation, which becomes prohibitively large for datasets with millions of rows. [BigGAM](../reference/BigGAM.md#whittaker.BigGAM) sidesteps this by discretizing each covariate into bins before fitting, reducing memory from O(n·p) to O(d·p) where d is the number of bins (typically 200 or fewer).


# Generate Data

Simulate a large dataset with two covariates and a response that combines a sinusoidal and a linear effect.


``` python
import numpy as np
import whittaker as wk

# Simulate large dataset with two covariates
rng = np.random.default_rng(23)
n = 500_000
x1 = rng.uniform(0, 2 * np.pi, n)
x2 = rng.uniform(0, 1, n)
y = np.sin(x1) + 0.5 * x2 + rng.normal(0, 0.3, n)
data = {"y": y, "x1": x1, "x2": x2}
```


# Fit

[BigGAM](../reference/BigGAM.md#whittaker.BigGAM) accepts the same formula syntax as a regular [GAM](../reference/GAM.md#whittaker.GAM). The default method is `"fREML"`, which is better suited to discretized data than `"REML"`.


``` python
model = wk.BigGAM("y ~ s(x1, k=10) + s(x2, k=10)").fit(data)
model.summary()
```


    GAM fit summary
    ============================================================
    Formula:    y ~ s(x1, k=10) + s(x2, k=10)
    Family:     Gaussian(link='identity')
    Inference:  REML
    Observations: 0
    Coefficients: 19

    Parametric coefficients:
      Term                       Estimate    Std.Err    t value    p-value
      ------------------------ ---------- ---------- ---------- ----------
      (Intercept)                  0.2503     0.0000      0.000          1

    Approximate significance of smooth terms:
      Term                        EDF Ref.df     Chi.sq    p-value
      ------------------------ ------ ------ ---------- ----------
      s(x1, k=10)                9.00      9 2783105.298    < 1e-16
      s(x2, k=10)                1.02      2 115841.852    < 1e-16

    Total EDF:  11.01
    Scale est:  0.089773
    Deviance:   44885.4985
    Null dev:   305380.4545
    Dev. expl:  85.3%
    GCV score:  0.089775
    AIC:        213713.90
    BIC:        213836.39


# Predict

Prediction works identically to a standard GAM: pass a dict of arrays for the covariates of interest.


``` python
new_data = {"x1": np.array([0.5, 1.0, 2.0]), "x2": np.array([0.2, 0.5, 0.8])}
preds = model.predict(new_data)
preds.values
```


    array([0.57799383, 1.09238007, 1.30912694])


# Interpret

[BigGAM](../reference/BigGAM.md#whittaker.BigGAM) discretizes each covariate into at most `n_discrete=200` unique bins, reducing the fitting problem from O(n·p) to O(d·p) where d ≪ n. The smooth estimates are essentially identical to a full GAM fitted on the raw data, because the precision gained from additional observations beyond the bin resolution is negligible. This makes [BigGAM](../reference/BigGAM.md#whittaker.BigGAM) a practical drop-in for [GAM](../reference/GAM.md#whittaker.GAM) when data is too large to hold a full design matrix in memory (typically when n exceeds a few hundred thousand rows).