eval.CustomMetric

A user-defined scoring metric.

Usage

Source

eval.CustomMetric(
    name,
    scorer_fn,
    description="",
)

Parameters

name: str

Unique metric name (e.g. "code_executes").

scorer_fn: Any

Callable receiving (query, response, context) and returning a float between 0.0 and 1.0.

description: str = ""

Human-readable description of what the metric measures.