Evaluate a survival model’s out-of-sample performance using k-fold cross-validation.
cross_validate(
model,
surv,
covariates,
*,
data=None,
k=5,
metric=None,
metrics=None,
times=None,
stratified=True,
seed=None,
)
Provides an honest, unbiased estimate of model performance by splitting data into folds, fitting on training folds, and evaluating on held-out test folds. This avoids overfitting bias that occurs when fitting and scoring on the same data.
Why cross-validate? Fitting and scoring on the training data gives overly optimistic performance estimates. A model may fit the training data well due to overfitting, not true predictive ability. Cross-validation repeatedly fits on different training splits and evaluates on held-out data, simulating performance on new subjects.
Metrics:
"concordance": Harrell’s C-statistic on the test fold. Higher is better (0.5 = random, 1.0 = perfect)."brier": Integrated IPCW Brier score over specified times. Lower is better (0 = perfect calibration, 1 = worst). Requires explicit times= parameter."auc": Integrated time-dependent AUC (Uno estimator). Higher is better (0.5 = random, 1.0 = perfect). Requires explicit times= parameter.
Parameters
model: Any-
An unfitted estimator instance (e.g., CoxPH(), CoxNet(), AFT("weibull"), RoystonParmar(df=3)). A fresh copy is fit on each training fold, leaving the passed object unchanged.
surv: Surv-
A Surv response (time-to-event data). Can be right-censored or counting-process. Weights in the response are carried through the cross-validation.
covariates: Any-
Covariates/predictors for the model. Can be:
- A 2-D array or pandas/Polars DataFrame with one row per subject
- A formula string (as in CoxPH.fit()), evaluated against
data
data: Any = None-
If covariates is a formula string, the data frame to evaluate it against.
k: int = 5-
Number of folds (default 5). Each fold serves as test data once; subjects are split randomly and evenly across folds. Typical choices: 5 or 10.
metric: str | None = None-
A single performance metric (backward-compatible). Use metrics instead to evaluate multiple metrics in a single CV run. If neither is provided, defaults to "concordance".
metrics: list[str] | None = None-
A list of metrics to evaluate in a single CV run. The model is fit once per fold and scored on all requested metrics. Cannot be used together with metric. Supported options are: "concordance", "brier", and "auc".
times: Any = None-
Evaluation time points for "brier" and "auc" metrics (1-D array-like, length \ge 2). Required when using those metrics. Example: times=[365, 730, 1095] for 1-, 2-, and 3-year predictions.
stratified: bool = True-
If True (default), use stratified k-fold ensuring balanced event/censoring representation across folds. This prevents singular matrix errors and biased CV estimates on imbalanced survival data (rare events). If False, use simple random k-fold shuffling.
seed: int | None = None-
Random seed for fold shuffling, ensures reproducibility. If
None, results may vary between runs. Use a fixed seed for consistent comparisons.
Returns
dict-
When metric (singular) is used, returns a flat dictionary:
"metric": Metric name used."k": Number of folds."scores": List of per-fold scores."mean": Mean score across folds."std": Standard deviation of scores.
When metrics (plural) is used, returns a keyed dictionary:
"k": Number of folds."results": Dict keyed by metric name, each with "scores", "mean", and "std".
Details
How folds work: By default (stratified=True), subjects are grouped by event status (censored vs. event, or multiple event types), then randomly shuffled within each stratum and split into k roughly equal-sized groups. This ensures each fold has approximately the same proportion of events and censored observations as the overall dataset. This is crucial for imbalanced data (e.g., rare events) to prevent singular matrix errors and ensures unbiased cross-validation estimates.
If stratified=False, subjects are simply shuffled and split randomly, which may lead to folds with very different event rates and can destabilize model fitting on sparse data.
Multi-metric mode: When metrics is a list, the model is fit once per fold and all requested metrics are computed on the same held-out data. This is more efficient than calling cross_validate separately for each metric (which would refit the model each time) and ensures all metrics see the same folds.
Completeness: Subjects with missing covariates are dropped before folding. This ensures all folds use the same cleaned data, avoiding alignment issues.
AFT model note: For AFT, concordance uses the negated linear predictor (since in AFT, larger lp means longer survival, opposite to Cox). This is handled automatically.
Reproducibility: Set seed= to ensure the same folds are used across runs. This is important for comparing different models or reporting consistent results.
Examples
Evaluate a Cox model with 5-fold cross-validation using concordance:
import greenwood as gw
# Load data and build a right-censored response
lung = gw.load_dataset("lung", backend="polars")
y = gw.Surv.right(lung["time"], event=(lung["status"] == 2))
# Run 5-fold cross-validation with concordance
result = gw.cross_validate(
gw.CoxPH(), y, lung[["age", "sex"]], k=5, metric="concordance", seed=1
)
result
{'metric': 'concordance',
'k': 5,
'scores': [0.5459697732997482,
0.6741645244215938,
0.5275779376498801,
0.696520618556701,
0.5741469816272966],
'mean': 0.603675967111044,
'std': 0.07678157638643948}
Access individual components. The mean concordance across folds:
# Mean concordance across folds
result["mean"]
Per-fold scores (variability check):
# Per-fold concordance scores
result["scores"]
[0.5459697732997482,
0.6741645244215938,
0.5275779376498801,
0.696520618556701,
0.5741469816272966]
Standard deviation (estimate of generalization uncertainty):
# Standard deviation of fold scores
result["std"]
Evaluate multiple metrics in a single CV run. The model is fit once per fold and scored on all requested metrics:
# Evaluate concordance, Brier, and AUC in one pass
result_multi = gw.cross_validate(
gw.CoxPH(), y, lung[["age", "sex"]], k=5,
metrics=["concordance", "brier", "auc"],
times=[180, 365, 540], seed=1
)
result_multi
{'k': 5,
'results': {'concordance': {'scores': [0.5459697732997482,
0.6741645244215938,
0.5275779376498801,
0.696520618556701,
0.5741469816272966],
'mean': 0.603675967111044,
'std': 0.07678157638643948},
'brier': {'scores': [0.2136207488479263,
0.23838812973571868,
0.23271598116657594,
0.19349410160275993,
0.21970890492256898],
'mean': 0.21958557325510997,
'std': 0.017623159308294348},
'auc': {'scores': [0.5819078127718599,
0.6578037795038837,
0.4953653913748597,
0.7303004762804867,
0.5968618566872875],
'mean': 0.6124478633236754,
'std': 0.087792825791058}}}
Access results for a specific metric:
# Mean concordance
result_multi["results"]["concordance"]["mean"]
# Mean integrated Brier score
result_multi["results"]["brier"]["mean"]