# Add a Random Effect

A random effect smooth `s(group_id, bs='re')` treats group-level intercepts as penalised deviations from the global mean. This is equivalent to a classical random intercept: groups with little data get pulled strongly toward the overall mean (shrinkage), while well-observed groups stay closer to their empirical mean. Use this when you have repeated observations on many subjects or clusters and want to account for between-group variation without overfitting.


# Generate data

Simulate 10 subjects, each with a different underlying intercept and a small number of observations. The true intercepts are drawn from a normal distribution, exactly what the random effect prior assumes.


``` python
import whittaker as wk
import numpy as np

# Set subject and observation counts
rng = np.random.default_rng(7)
n_subjects = 10
obs_per_subject = 8
n = n_subjects * obs_per_subject

# Draw random subject intercepts and predictors
subject_id = np.repeat(np.arange(n_subjects), obs_per_subject)
subject_intercepts = rng.normal(0, 2, size=n_subjects)
x = rng.uniform(0, 5, size=n)

# Simulate response with linear trend and subject offsets
y = (
    2 + 0.5 * x
    + subject_intercepts[subject_id]
    + rng.normal(0, 0.5, size=n)
)

data = {"y": y, "x": x, "subject_id": subject_id}
```


Each subject has its own random offset on top of a linear trend. Without accounting for this grouping, the residuals would be correlated within subjects.


# Fit

Fit a model with a linear effect of `x` and a random effect for `subject_id`.


``` python
# Fit linear trend with random subject intercepts
model = wk.GAM("y ~ x + s(subject_id, bs='re')").fit(data)

model.summary()
```


    GAM fit summary
    ============================================================
    Formula:    y ~ x + s(subject_id, bs='re')
    Family:     Gaussian(link='identity')
    Inference:  GCV
    Observations: 80
    Coefficients: 11

    Parametric coefficients:
      Term                       Estimate    Std.Err    t value    p-value
      ------------------------ ---------- ---------- ---------- ----------
      (Intercept)                  1.7729     0.1002     17.700    < 1e-16
      x                            0.3901     0.0358     10.892  1.198e-16

    Approximate significance of smooth terms:
      Term                        EDF Ref.df     Chi.sq    p-value
      ------------------------ ------ ------ ---------- ----------
      s(subject_id, bs='re')     8.88      9    690.509    < 1e-16

    Total EDF:  10.88
    Scale est:  0.191451
    Deviance:   13.2325
    Null dev:   177.3253
    Dev. expl:  92.5%
    GCV score:  0.221597
    AIC:        105.66
    BIC:        131.59


The random effect appears as a single smooth term in the summary. Its EDF reflects how much variation exists across groups relative to the penalty. More between-subject variation leads to a higher EDF (less shrinkage). Check that the linear term for `x` has a coefficient close to the true value of 0.5.


# Partial effects


``` python
wk.partial_effects(model)
```


<style>
  #altair-viz-9a6f62f351e248298f16d93b193fa85f.vega-embed {
    width: 100%;
    display: flex;
  }

  #altair-viz-9a6f62f351e248298f16d93b193fa85f.vega-embed details,
  #altair-viz-9a6f62f351e248298f16d93b193fa85f.vega-embed details summary {
    position: relative;
  }
</style>


The partial effects plot for `x` shows the linear trend. The random effect panel shows the estimated subject-level deviations, each shrunk toward zero. Subjects with fewer observations or noisier data will be pulled more strongly toward the centre.


# Diagnostics


``` python
wk.check(model)
```


<style>
  #altair-viz-d119f0a8a8414d44920be1dc03733754.vega-embed {
    width: 100%;
    display: flex;
  }

  #altair-viz-d119f0a8a8414d44920be1dc03733754.vega-embed details,
  #altair-viz-d119f0a8a8414d44920be1dc03733754.vega-embed details summary {
    position: relative;
  }
</style>


With a correctly specified random effect, the Q-Q plot should be straight and the residuals should show no remaining within-subject pattern. Systematic curvature in the residual plots would suggest the linear `x` term needs to be replaced with a smooth.