cross_validate()

Evaluate a survival model’s out-of-sample performance using k-fold cross-validation.

Usage

Source

cross_validate(
    model,
    surv,
    covariates,
    *,
    data=None,
    k=5,
    metric="concordance",
    times=None,
    seed=None
)

Provides an honest, unbiased estimate of model performance by splitting data into folds, fitting on training folds, and evaluating on held-out test folds. This avoids overfitting bias that occurs when fitting and scoring on the same data.

Why cross-validate? Fitting and scoring on the training data gives overly optimistic performance estimates. A model may fit the training data well due to overfitting, not true predictive ability. Cross-validation repeatedly fits on different training splits and evaluates on held-out data, simulating performance on new subjects.

Metrics:

  • "concordance" (default): Harrell’s C-statistic on the test fold. Higher is better (0.5 = random, 1.0 = perfect). Requires CoxPH, CoxNet, or AFT model.
  • "brier": Integrated IPCW Brier score over specified times. Lower is better (0 = perfect calibration, 1 = worst). Requires explicit times= parameter.

Parameters

model: Any

An unfitted estimator instance (e.g., CoxPH(), CoxNet(), AFT("weibull")). A fresh copy is fit on each training fold, leaving the passed object unchanged. Supported: CoxPH, CoxNet, AFT (for concordance) and any of those (for Brier).

surv: Surv

A Surv response (time-to-event data). Can be right-censored or counting-process. Weights in the response are carried through the cross-validation.

covariates: Any

Covariates/predictors for the model. Can be:

  • A 2-D array or pandas/Polars DataFrame with one row per subject
  • A formula string (as in CoxPH.fit()), evaluated against data
data: Any = None

If covariates is a formula string, the data frame to evaluate it against.

k: int = 5

Number of folds (default 5). Each fold serves as test data once; subjects are split randomly and evenly across folds. Typical choices: 5 or 10.

metric: str = "concordance"

Performance metric for evaluation:

  • "concordance" (default): Harrell’s C-statistic. Requires CoxPH, CoxNet, or AFT.
  • "brier": Integrated inverse-probability-of-censoring-weighted (IPCW) Brier score. Requires times= with at least 2 time points.
times: Any = None

For metric="brier", evaluation time points (1-D array-like, length \(\ge 2\)). The Brier score is computed at each time, then integrated (time-averaged). Example: times=[365, 730, 1095] for 1, 2, 3-year predictions.

seed: int | None = None
Random seed for fold shuffling, ensures reproducibility. If None, results may vary between runs. Use a fixed seed for consistent comparisons.

Returns

dict

Dictionary with keys:

  • "metric": Metric name used ("concordance" or "brier").
  • "k": Number of folds.
  • "scores": List of per-fold scores (one per fold).
  • "mean": Mean score across folds (primary summary).
  • "std": Standard deviation of scores (variability estimate).
For concordance, higher mean is better. For Brier, lower mean is better.

Details

How folds work: Subjects are randomly shuffled and split into k roughly equal-sized groups. On iteration i, fold i is held out for testing, while the other k-1 folds are combined for training. This repeats k times until each fold has served as test data once.

Completeness: Subjects with missing covariates are dropped before folding. This ensures all folds use the same cleaned data, avoiding alignment issues.

AFT model note: For AFT, concordance uses the negated linear predictor (since in AFT, larger lp means longer survival, opposite to Cox). This is handled automatically.

Reproducibility: Set seed= to ensure the same folds are used across runs. This is important for comparing different models or reporting consistent results.

Examples

Evaluate a Cox model with 5-fold cross-validation using concordance:

import greenwood as gw

lung = gw.load_dataset("lung", backend="polars")
y = gw.Surv.right(lung["time"], event=(lung["status"] == 2))
result = gw.cross_validate(
    gw.CoxPH(), y, lung[["age", "sex"]], k=5, metric="concordance", seed=1
)
result
{'metric': 'concordance',
 'k': 5,
 'scores': [0.5720524017467249,
  0.6159147869674185,
  0.44519704433497537,
  0.6560468140442133,
  0.687603305785124],
 'mean': 0.5953628705756911,
 'std': 0.09448065076706443}

Access individual components. The mean concordance across folds:

result["mean"]
0.5953628705756911

Per-fold scores (variability check):

result["scores"]
[0.5720524017467249,
 0.6159147869674185,
 0.44519704433497537,
 0.6560468140442133,
 0.687603305785124]

Standard deviation (estimate of generalization uncertainty):

result["std"]
0.09448065076706443

Use Brier score (calibration) instead of concordance (discrimination):

result_brier = gw.cross_validate(
    gw.CoxPH(), y, lung[["age", "sex"]], k=5,
    metric="brier", times=[180, 365, 540], seed=1
)
result_brier
{'metric': 'brier',
 'k': 5,
 'scores': [0.22789930822784574,
  0.2467842310902283,
  0.23420131250618006,
  0.20813041960234802,
  0.19010143100919175],
 'mean': 0.22142334048715878,
 'std': 0.022395051117791446}

Compare two models via cross-validation. Model with higher mean concordance (or lower mean Brier) generalizes better:

# simple_model = gw.CoxPH()
# complex_model = gw.CoxPH()
# simple_cv = gw.cross_validate(simple_model, y, lung[["age"]], seed=1)
# complex_cv = gw.cross_validate(complex_model, y, lung[["age", "sex", "ph.ecog"]], seed=1)
# print(f"Simple model C-index: {simple_cv['mean']:.3f} ± {simple_cv['std']:.3f}")
# print(f"Complex model C-index: {complex_cv['mean']:.3f} ± {complex_cv['std']:.3f}")