conformal_coverage()

Compute empirical coverage of conformal intervals on held-out data.

Usage

Source

conformal_coverage(
    predictor,
    data,
    response,
)

A useful sanity check that the realized coverage on a given dataset is close to (at least) the nominal predictor.level; systematic under-coverage may indicate a violation of the exchangeability assumption underlying conformal prediction (e.g. distribution shift between calibration and test data).

Parameters

predictor: ConformalPredictor

A fitted ConformalPredictor.

data: InputData

Data containing both covariates and the response.

response: str

Name of the response variable.

Returns

float

Fraction of observations falling within the conformal interval.