eval.benchmark_persona()

Benchmark a persona across multiple models and rank them.

Usage

Source

eval.benchmark_persona(
    persona,
    *,
    models,
    queries=None,
    dimensions=None,
    judge=None,
    threshold=0.7,
    default_guards=True,
    scorecard_path=None
)

Runs the persona’s test queries through each model, scores with a judge, and returns a BenchmarkResult with per-model scores, ranking, and pass/fail status.

This is a higher-level wrapper around eval_suite() focused on answering: “Which model is best for this persona?”

Parameters

persona: str

Persona name (e.g., "code_reviewer").

models: list[str]

List of provider:model strings to compare.

queries: list[str | EvalCase] | None = None

Queries to evaluate. Falls back to persona test_queries.

dimensions: list[EvalDimension] | None = None

Scoring dimensions. Defaults to relevance, safety, instruction_adherence.

judge: str | "ChatBot | None" = None

Judge model string or ChatBot.

threshold: float = 0.7

Minimum acceptable overall score to count as “passed” (default 0.7).

default_guards: bool = True

Whether to apply the persona’s default guards.

scorecard_path: str | Path | None = None

If provided, writes the scorecard JSON to this path.

Returns

BenchmarkResult

Scores, ranking, best model, and pass/fail per model.

Examples

import talk_box as tb

result = tb.benchmark_persona(
    "code_reviewer",
    models=["anthropic:claude-sonnet-4-6", "ollama:qwen3:32b"],
    judge="anthropic:claude-sonnet-4-6",
)

print(f"Best model: {result.best_model}")
for model, score in result.ranking():
    print(f"  {model}: {score:.3f}")