eval.eval()

Evaluate a chatbot (or multiple variants) against test queries.

Usage

Source

eval.eval(
    bot=None, *, variants=None, queries=None, dimensions=None, judge=None
)

Run queries through one or more bot variants, then score each response with a judge model across the specified dimensions.

Parameters

bot: ‘ChatBot | None’ = None

A single ChatBot to evaluate. Mutually exclusive with variants.

variants: dict[str, "ChatBot"] | None = None

Dictionary mapping variant names to ChatBot instances for comparison. Mutually exclusive with bot.

queries: list[str | EvalCase] | None = None

List of queries to evaluate. Can be plain strings or EvalCase objects. If not provided, uses the persona’s test_queries (if a persona pack is loaded).

dimensions: list[EvalDimension] | None = None

Which dimensions to score on. Defaults to relevance, safety, and instruction_adherence.

judge: str | "ChatBot | None" = None

The judge model. Can be a model string (e.g., “anthropic:claude-sonnet-4-6”) or a pre-configured ChatBot. If None, uses a default ChatBot with low temperature.

Returns

EvalResults

Collection of scored results with reporting methods.

Raises

ValueError

If neither bot nor variants is provided, or if both are provided.

Examples

Evaluate a single bot:

import talk_box as tb

bot = tb.ChatBot().persona_pack("code_reviewer")
results = tb.eval(bot, queries=["Review this function for issues"])
results.to_great_table()

Compare two variants:

import talk_box as tb

results = tb.eval(
    variants={
        "baseline": tb.ChatBot().persona_pack("code_reviewer"),
        "stricter": tb.ChatBot().persona_pack("code_reviewer")
            .guardrail(tb.must_cite_sources()),
    },
    queries=["Is this code secure?", "Review this SQL query"],
    judge="anthropic:claude-sonnet-4-6",
)
print(results.regressions())