eval.EvalResults
Collection of evaluation results with reporting capabilities.
Usage
eval.EvalResults()Parameters
results: list[EvalResult] = list()-
List of individual eval results.
config: dict[str, Any] = dict()- Metadata about the evaluation run.
Attributes
| Name | Description |
|---|---|
| dimensions | Unique dimensions scored across all results. |
| variants | Unique variant names in the results. |
dimensions
Unique dimensions scored across all results.
dimensions: list[EvalDimension | str]
variants
Unique variant names in the results.
variants: list[str]
Methods
| Name | Description |
|---|---|
| passed() | Check if all variants meet the minimum threshold. |
| regressions() | Detect regressions between variants. |
| scores_by_variant() | Aggregate mean scores per variant per dimension. |
| summary() | Compute summary statistics for the eval run. |
| to_dataframe() | Export results to a pandas DataFrame. |
| to_great_table() | Create a Great Tables comparison report. |
| to_scorecard() | Export results as a scorecard dictionary (optionally written to JSON). |
passed()
Check if all variants meet the minimum threshold.
Usage
passed(threshold=0.7)Parameters
threshold: float = 0.7- Minimum acceptable average score (0.0 to 1.0).
Returns
bool- True if all variants have an overall score >= threshold.
regressions()
Detect regressions between variants.
Usage
regressions(baseline=None, threshold=0.05)Compares each variant to the baseline and returns dimensions where the score dropped by more than threshold.
Parameters
baseline: str | None = None-
Variant name to use as baseline. Defaults to the first variant.
threshold: float = 0.05- Minimum score drop to flag as a regression.
Returns
dict[str, dict[str, float]]- Mapping of variant name -> dimension -> score delta (negative = regression).
scores_by_variant()
Aggregate mean scores per variant per dimension.
Usage
scores_by_variant()Returns
dict[str, dict[str, float]]- Mapping of variant name -> dimension name -> mean score.
summary()
Compute summary statistics for the eval run.
Usage
summary()Returns
dict[str, Any]- Summary with total queries, variants, dimension means, and overall scores.
to_dataframe()
Export results to a pandas DataFrame.
Usage
to_dataframe()Returns
pd.DataFrame- DataFrame with one row per (variant, query, dimension) combination.
Raises
ImportError- If pandas is not installed.
to_great_table()
Create a Great Tables comparison report.
Usage
to_great_table()Produces a summary table showing mean scores per variant per dimension, with color-coded cells indicating quality levels.
Returns
gt.GT- A formatted Great Tables object ready for display or export.
Raises
ImportError- If great_tables or pandas are not installed.
to_scorecard()
Export results as a scorecard dictionary (optionally written to JSON).
Usage
to_scorecard(path=None)The scorecard is a portable representation of evaluation results suitable for committing to a repository or publishing to a docs site.
Parameters
path: str | Path | None = None- Optional file path to write the scorecard JSON. Directories are created automatically.
Returns
dict[str, Any]- Scorecard with metadata, per-variant scores, and overall results.