eval.artifact_correctness_scorer()
Score a response’s code artifacts for correctness.
Usage
eval.artifact_correctness_scorer(
query, response, context="", *, expected_output=None, timeout=10.0
)Extracts Python code blocks from the response, executes each in an isolated subprocess, and returns a score based on:
- 1.0 — all code blocks execute successfully (and match expected output if provided)
- 0.5 — code executes but output doesn’t match expected
- 0.0 — code fails to execute or no code blocks found
Parameters
query: str-
The original query/prompt.
response: str-
The model’s response containing code blocks.
context: str = ""-
Optional additional context (unused, kept for scorer API compat).
expected_output: str | None = None-
If provided, the stdout of the code must contain this string (case-insensitive) to score above 0.5.
timeout: float = 10.0- Maximum seconds to allow each code block to run.
Returns
float- Score between 0.0 and 1.0.