# Handle Overdispersed Counts with Negative Binomial

Count data often violates the Poisson assumption that the variance equals the mean. When the observed variance is substantially larger (a condition called overdispersion) a Poisson GAM underestimates uncertainty and produces confidence intervals that are too narrow. The Negative Binomial family adds a dispersion parameter that absorbs the extra variation, giving honest standard errors and better-calibrated predictions.


# Load data

Load the fish dataset and inspect it.


``` python
import whittaker as wk

data = wk.load_dataset("fish", as_frame=True)
data.columns.tolist()
```


    ['temperature', 'depth', 'count']


``` python
data.head()
```


|     | temperature | depth     | count |
|-----|-------------|-----------|-------|
| 0   | 13.232243   | 20.213893 | 7.0   |
| 1   | 13.969823   | 20.798765 | 14.0  |
| 2   | 24.284515   | 36.965959 | 3.0   |
| 3   | 9.838319    | 30.989895 | 2.0   |
| 4   | 20.002011   | 45.968733 | 4.0   |


# Fit a Poisson model first

Start with Poisson to establish a baseline and check for overdispersion.


``` python
poisson_model = wk.GAM(
    "count ~ s(temperature) + s(depth)",
    family=wk.Poisson(),
).fit(data)

poisson_model.summary()
```


    GAM fit summary
    ============================================================
    Formula:    count ~ s(temperature) + s(depth)
    Family:     Poisson(link='log')
    Inference:  GCV
    Observations: 300
    Coefficients: 19

    Parametric coefficients:
      Term                       Estimate    Std.Err    z value    p-value
      ------------------------ ---------- ---------- ---------- ----------
      (Intercept)                  1.5349     0.0289     53.128    < 1e-16

    Approximate significance of smooth terms:
      Term                        EDF Ref.df     Chi.sq    p-value
      ------------------------ ------ ------ ---------- ----------
      s(temperature)             4.24      5    294.642    < 1e-16
      s(depth)                   1.74      2    194.880    < 1e-16

    Total EDF:  6.98
    Scale est:  1.000000
    Deviance:   320.5707
    Null dev:   843.4578
    Dev. expl:  62.0%
    GCV score:  1.120093
    AIC:        1305.79
    BIC:        1331.65


Look at the summary's residual deviance relative to the residual degrees of freedom. A ratio well above 1 (sometimes called the dispersion ratio) signals overdispersion. When the ratio is large, Poisson confidence intervals are too narrow and p-values are anti-conservative.


# Fit a Negative Binomial model

Refit with `wk.NegativeBinomial()`, which estimates the extra dispersion parameter automatically.


``` python
nb_model = wk.GAM(
    "count ~ s(temperature) + s(depth)",
    family=wk.NegativeBinomial(),
).fit(data)

nb_model.summary()
```


    GAM fit summary
    ============================================================
    Formula:    count ~ s(temperature) + s(depth)
    Family:     NegativeBinomial(theta=329.8, link='log')
    Inference:  GCV
    Observations: 300
    Coefficients: 19

    Parametric coefficients:
      Term                       Estimate    Std.Err    z value    p-value
      ------------------------ ---------- ---------- ---------- ----------
      (Intercept)                  1.5347     0.0291     52.789    < 1e-16

    Approximate significance of smooth terms:
      Term                        EDF Ref.df     Chi.sq    p-value
      ------------------------ ------ ------ ---------- ----------
      s(temperature)             4.24      5    290.499    < 1e-16
      s(depth)                   1.73      2    190.878    < 1e-16

    Total EDF:  6.97
    Scale est:  1.000000
    Deviance:   315.5890
    Null dev:   829.7217
    Dev. expl:  62.0%
    GCV score:  1.102575
    AIC:        1305.71
    BIC:        1331.51


Compare the deviance explained between the two models. The Negative Binomial typically shows a lower (more honest) deviance explained because it no longer tries to explain variance that is simply inherent noise. The estimated dispersion parameter appears in the summary. A value near zero indicates little overdispersion, a large value indicates substantial overdispersion.


# Partial effects


``` python
wk.partial_effects(nb_model)
```


<style>
  #altair-viz-09482460c36b4b10822a8e5724e88e82.vega-embed {
    width: 100%;
    display: flex;
  }

  #altair-viz-09482460c36b4b10822a8e5724e88e82.vega-embed details,
  #altair-viz-09482460c36b4b10822a8e5724e88e82.vega-embed details summary {
    position: relative;
  }
</style>


The partial effect plots show the relationship between each covariate and the log-expected count. Confidence bands will generally be wider than those from the Poisson model, reflecting the additional uncertainty that overdispersion introduces.


# Diagnostics


``` python
wk.check(nb_model)
```


<style>
  #altair-viz-e941eb538a9947d5a5b8a8e58ba707fc.vega-embed {
    width: 100%;
    display: flex;
  }

  #altair-viz-e941eb538a9947d5a5b8a8e58ba707fc.vega-embed details,
  #altair-viz-e941eb538a9947d5a5b8a8e58ba707fc.vega-embed details summary {
    position: relative;
  }
</style>


A well-fitting Negative Binomial model should show residuals with no strong pattern against fitted values, and a Q-Q plot closer to the diagonal than the Poisson model produced.