# PolarsGAM


GAM that reads data from Polars LazyFrames, DataFrames, or files.


Usage

``` python
PolarsGAM(
    formula,
    family=None,
    *,
    n_discrete=200,
    chunk_size=100000,
)
```


[PolarsGAM](PolarsGAM.md#whittaker.PolarsGAM) extends `~whittaker.bam.BigGAM` with a data-loading layer built on Polars, so it can source data from an in-memory Polars `DataFrame`/`LazyFrame` or directly from a file on disk (CSV, Parquet, IPC/Arrow, or NDJSON), without requiring the caller to materialize the whole dataset into a `dict` of NumPy arrays first. File paths are opened lazily (`pl.scan_*`), and the resulting `LazyFrame` is collected using Polars' streaming query engine (`collect(streaming=True)`), which evaluates the query plan incrementally rather than loading the entire source at once. The collected frame is then walked in `chunk_size`-row slices (`iter_slices`) and each column is converted to a NumPy array and concatenated, producing the same `dict[str, numpy.ndarray]` that `~whittaker.gam.GAM.fit` and `~whittaker.bam.BigGAM.fit` expect. Fitting itself then proceeds exactly as in [BigGAM](BigGAM.md#whittaker.BigGAM): covariates are discretized to at most [n_discrete](BigGAM.md#whittaker.BigGAM.n_discrete) unique values per smooth, and the design matrix is never materialized.

Use [PolarsGAM](PolarsGAM.md#whittaker.PolarsGAM) when the data already lives in Polars, or on disk in a Polars-readable format, and you want to avoid a manual load-then-convert step -- particularly for datasets in the 1M-100M row range where Polars' streaming engine keeps peak memory bounded during the read. For SQL-native sources or datasets that are more naturally expressed as a database query (joins, filters, aggregations), see `~whittaker.duckdb.DuckDBGAM` instead.

Requires the `polars` package (install via `pip install whittaker[polars]`).


## Parameters


`formula: str or Formula`  
Model formula as a string (e.g. `"y ~ s(x1) + s(x2) + x3"`), or an already-parsed [Formula](Formula.md#whittaker.Formula) object. Same syntax as `~whittaker.gam.GAM`.

`family: Family = None`  
Response distribution family, e.g. [Gaussian()](Gaussian.md#whittaker.Gaussian), [Binomial()](Binomial.md#whittaker.Binomial), [Poisson()](Poisson.md#whittaker.Poisson), [Gamma()](Gamma.md#whittaker.Gamma), or [Tweedie()](Tweedie.md#whittaker.Tweedie). Defaults to [Gaussian()](Gaussian.md#whittaker.Gaussian).

`n_discrete: int = ``200`  
Maximum number of unique representative values per covariate used when discretizing smooth terms (see `~whittaker.bam.BigGAM`). Defaults to `200`.

`chunk_size: int = ``100000`  
Number of rows collected per slice when converting the source into NumPy arrays. Smaller values reduce peak memory during the Polars-to-NumPy conversion step at the cost of more Python-level overhead; larger values reduce overhead but require more memory per slice. Defaults to `100_000`.


## Examples


``` python
import numpy as np
import polars as pl
from whittaker.polars_streaming import PolarsGAM

rng = np.random.default_rng(0)
n = 5_000
x1 = rng.uniform(0, 1, n)
x2 = rng.uniform(0, 1, n)
y = np.sin(2 * np.pi * x1) + x2**2 + rng.normal(scale=0.2, size=n)

df = pl.DataFrame({"x1": x1, "x2": x2, "y": y})

model = PolarsGAM("y ~ s(x1) + s(x2)", n_discrete=100, chunk_size=1_000)
model.fit(df.lazy())
print(model.summary())
```


    GAM fit summary
    ============================================================
    Formula:    y ~ s(x1) + s(x2)
    Family:     Gaussian(link='identity')
    Inference:  REML
    Observations: 0
    Coefficients: 19

    Parametric coefficients:
      Term                       Estimate    Std.Err    t value    p-value
      ------------------------ ---------- ---------- ---------- ----------
      (Intercept)                  0.3376     0.0000      0.000          1

    Approximate significance of smooth terms:
      Term                        EDF Ref.df     Chi.sq    p-value
      ------------------------ ------ ------ ---------- ----------
      s(x1)                      8.91      9  62254.971    < 1e-16
      s(x2)                      6.26      7  11059.123    < 1e-16

    Total EDF:  16.17
    Scale est:  0.040068
    Deviance:   199.6941
    Null dev:   3148.4427
    Dev. expl:  93.7%
    GCV score:  0.040198
    AIC:        -1880.28
    BIC:        -1774.91


A file path can be passed directly instead of a `DataFrame`/`LazyFrame` -- [PolarsGAM](PolarsGAM.md#whittaker.PolarsGAM) infers the format from the extension and scans it lazily:


``` python
model = PolarsGAM("y ~ s(x)")
model.fit("large_dataset.parquet")
```


Fitting from an actual multi-gigabyte file requires `pip install whittaker[polars]` and enough disk I/O bandwidth to stream the file; the in-memory example above is kept small so it runs quickly, but the same code path scales to files with tens of millions of rows.


## Attributes

| Name | Description |
|----|----|
| [chunk_size](#chunk_size) | Chunk size used when converting the Polars source to NumPy arrays. |
| [n_rows](#n_rows) | Total number of rows in the source used by the most recent fit. |

------------------------------------------------------------------------


### chunk_size


Chunk size used when converting the Polars source to NumPy arrays.


`chunk_size: int`


This is the `chunk_size` value passed to `__init__`: the number of rows per slice that `_lazyframe_to_dict` requests from `LazyFrame.iter_slices` while converting the collected frame into the `dict[str, numpy.ndarray]` consumed by fitting.


------------------------------------------------------------------------


### n_rows


Total number of rows in the source used by the most recent fit.


`n_rows: int`


Populated by `fit()` via `_count_lazy`, which evaluates `lf.select(pl.len())` with Polars' streaming engine before collecting the data. Remains `0` until `fit()` has been called at least once.


## Methods

| Name | Description |
|----|----|
| [fit()](#fit) | Fit the GAM from a Polars source. |

------------------------------------------------------------------------


### fit()


Fit the GAM from a Polars source.


Usage

``` python
fit(
    data,
    *,
    smoothing_params=None,
    method="fREML",
    select=False,
    **kwargs,
)
```


Converts `data` to a Polars `LazyFrame`, collects it via Polars' streaming engine in `chunk_size`-row slices, and combines the result into `dict[str, numpy.ndarray]` before delegating to `~whittaker.bam.BigGAM`'s discretized fitting (see the class docstring for details on the discretization and cross-product accumulation).


#### Parameters


`data: polars.LazyFrame or polars.DataFrame or str or pathlib.Path`  
The data source. A Polars `LazyFrame` is used as-is; a `DataFrame` is converted with `.lazy()`; a file path (string or `Path`) is scanned lazily based on its extension -- `.parquet` (`scan_parquet`), `.csv` (`scan_csv`), `.ipc`/`.arrow` (`scan_ipc`), or `.ndjson`/`.jsonl` (`scan_ndjson`). Any other type or unrecognized extension raises `TypeError` or `ValueError` respectively.

`smoothing_params: list of float = None`  
Fixed smoothing parameters `lambda_j`, one per smooth term, in formula order. If `None` (the default), smoothing parameters are selected automatically according to [method](CausalGAM.md#whittaker.CausalGAM.method).

`method: str = ``"fREML"`  
Criterion used to select smoothing parameters when `smoothing_params` is `None`. One of `"fREML"` (default), `"REML"`, `"ML"`, or `"GCV"`. See `~whittaker.gam.GAM.fit` for a description of each criterion.

`select: bool = ``False`  
If `True`, add an extra penalty on each smooth's null space so the term can be shrunk to exactly zero (double-penalty selection). Defaults to `False`.

`**kwargs`  
Reserved for future keyword arguments; currently unused.


#### Returns


`PolarsGAM`  
Returns `self` for method chaining, e.g. `model = PolarsGAM("y ~ s(x)").fit(data)`.