Use a PyArrow Table

Pass a PyArrow Table directly to fit() without any manual conversion.

Whittaker uses Narwhals for data ingestion, which means that fit() accepts any DataFrame format Narwhals supports (Pandas, Polars, and PyArrow Tables) all without any conversion step on your part. This is useful when data arrives from Arrow-native sources such as Parquet files, Flight streams, or DuckDB queries.

Build a PyArrow Table

Load the wages dataset as a plain dict of NumPy arrays, then wrap it in pa.table(). The resulting PyArrow Table can be passed directly to fit().

import pyarrow as pa
import whittaker as wk

# Load dataset and wrap in a PyArrow Table
data = wk.load_dataset("wages")
table = pa.table(data)

Inspect the schema to confirm the columns and their Arrow types.

table.schema

age: double
experience: double
wage: double

Fit using the PyArrow Table

Pass the table to fit() exactly as you would a dict or a Pandas DataFrame. Narwhals handles the translation internally so no .to_pandas() or .to_pydict() call is needed.

model = wk.GAM("wage ~ s(age) + s(experience)", family=wk.Gamma()).fit(table)
model.summary()

GAM fit summary
============================================================
Formula:    wage ~ s(age) + s(experience)
Family:     Gamma(link='log')
Inference:  GCV
Observations: 800
Coefficients: 19

Parametric coefficients:
  Term                       Estimate    Std.Err    t value    p-value
  ------------------------ ---------- ---------- ---------- ----------
  (Intercept)                  3.7109     0.0084    440.686    < 1e-16

Approximate significance of smooth terms:
  Term                        EDF Ref.df     Chi.sq    p-value
  ------------------------ ------ ------ ---------- ----------
  s(age)                     3.23      4   2473.551    < 1e-16
  s(experience)              2.82      3    177.907    < 1e-16

Total EDF:  7.05
Scale est:  0.056726
Deviance:   44.9806
Null dev:   205.9584
Dev. expl:  78.2%
GCV score:  0.057230
AIC:        5881.68
BIC:        5914.71

Inspect model outputs

The fitted model is identical to one trained on a dict or a Polars DataFrame. Effective degrees of freedom and deviance explained are unaffected by the data format used at fit time.

model.edf_total

7.0508328482675875

model.deviance_explained

0.7816033886263458

From an Arrow-native source

In practice, PyArrow Tables often come from Parquet files or DuckDB result sets. The pattern is the same: no intermediate conversion is required.

# import duckdb
# table = duckdb.sql("SELECT age, experience, wage FROM 'wages.parquet'").arrow()
# model = wk.GAM("wage ~ s(age) + s(experience)", family=wk.Gamma()).fit(table)

The column names in the Arrow Table become the predictor names used in the formula. Whittaker reads only the columns referenced in the formula, so the table may contain additional columns without error.