Use a Polars DataFrame

Pass a Polars DataFrame directly to fit() without any manual conversion.

Whittaker uses Narwhals for data ingestion. This means that Polars DataFrames are accepted by fit() natively. And any DataFrame type that Narwhals supports (Pandas, PyArrow, cuDF, Modin) follows the same pattern.

From load_dataset

The load_dataset() function returns a plain dict of NumPy arrays. You can wrap it in pl.DataFrame and the result feeds directly into fit().

import polars as pl
import whittaker as wk

# Load dataset and convert to Polars DataFrame
data = wk.load_dataset("wages")
df = pl.DataFrame(data)

Check the dimensions of the resulting DataFrame to confirm the data loaded correctly.

df.shape

(800, 3)

The tuple shows the dataset has three columns: age, experience, and wage. Inspect the first few rows to verify the layout.

df.head(3)

shape: (3, 3)

age	experience	wage
f64	f64	f64
62.323637	9.216817	80.231223
42.032395	2.487841	40.024189
63.883454	7.251912	60.292852

Now fit a Gamma GAM using the Polars DataFrame directly.

# Fit Gamma GAM with two smooth predictors
model = wk.GAM("wage ~ s(age) + s(experience)", family=wk.Gamma()).fit(df)
model.summary()

GAM fit summary
============================================================
Formula:    wage ~ s(age) + s(experience)
Family:     Gamma(link='log')
Inference:  GCV
Observations: 800
Coefficients: 19

Parametric coefficients:
  Term                       Estimate    Std.Err    t value    p-value
  ------------------------ ---------- ---------- ---------- ----------
  (Intercept)                  3.7109     0.0084    440.686    < 1e-16

Approximate significance of smooth terms:
  Term                        EDF Ref.df     Chi.sq    p-value
  ------------------------ ------ ------ ---------- ----------
  s(age)                     3.23      4   2473.551    < 1e-16
  s(experience)              2.82      3    177.907    < 1e-16

Total EDF:  7.05
Scale est:  0.056726
Deviance:   44.9806
Null dev:   205.9584
Dev. expl:  78.2%
GCV score:  0.057230
AIC:        5881.68
BIC:        5914.71

The effective degrees of freedom gives a sense of the model’s overall complexity.

model.edf_total

7.0508328482675875

The deviance explained measures how well the two-predictor model accounts for variation in wage.

model.deviance_explained

0.7816033886263458

From a file

Reading a CSV with Polars and passing the result straight to fit() requires no intermediate steps.

# df = pl.read_csv("data.csv")
# model = wk.GAM("y ~ s(x1) + s(x2)").fit(df)

The column names in the CSV become the predictor names used in the formula. Whittaker reads only the columns it needs, so the file may contain extra columns without causing an error.

LazyFrame

Polars LazyFrame objects (created by pl.scan_csv(), pl.scan_parquet(), and similar scan functions) are not directly supported by GAM.fit(). Collect the frame first:

# lazy = pl.scan_csv("large_file.csv")
# df = lazy.collect()          # collect before passing to fit()
# model = wk.GAM("y ~ s(x)").fit(df)

For very large datasets that do not fit in memory, see the user guide section on PolarsGAM, which processes data in batches without loading it all at once.

Column types

Polars uses strict dtypes. Whittaker expects numeric predictors and the response to be floating-point or integer columns. If a column is stored as Utf8 (string) or Categorical, cast it before fitting.

# Cast column type and refit single-predictor model
df_cast = df.with_columns(pl.col("age").cast(pl.Float64))
model2 = wk.GAM("wage ~ s(age)", family=wk.Gamma()).fit(df_cast)

The simpler single-predictor model has fewer effective degrees of freedom than the two-predictor version.

model2.edf_total

4.5433512313739906

Deviance explained is also lower, as the experience predictor has been dropped from the formula.

model2.deviance_explained

0.7322978570125995