eval.eval_metric()

Register a custom evaluation metric.

Usage

Source

eval.eval_metric(
    name,
    scorer_fn,
    *,
    description="",
)

The scorer_fn receives (query, response, context) and must return a float between 0.0 and 1.0. Custom metrics participate in EvalResults.passed(), .regressions(), and scorecard rendering alongside the built-in judge dimensions.

Parameters

name: str

Unique metric identifier (e.g. "code_executes").

scorer_fn: Any

Scoring callable (query: str, response: str, context: str) -> float.

description: str = ""

Optional human-readable description.

Returns

CustomMetric

The registered metric object.

Examples

>>> def code_runs(query, response, context):
...     return 1.0 if "def " in response else 0.0
>>> tb.eval_metric("code_executes", code_runs)